[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:17 dist-test-slave-mnfb krb5kdc[4300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:17 dist-test-slave-mnfb krb5kdc[4300](info): set up 2 sockets
Sep 05 07:53:17 dist-test-slave-mnfb krb5kdc[4300](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250905 07:53:19.434258 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:19 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:19.575742 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44457
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:44457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:19.683310 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:19.683578 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:19.683634 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:19.686997 4316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:19.687089 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:19.687111 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:19.687129 4316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:19.687145 4316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:19.691676 4316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:44457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:19.692883 4316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:19.693864 4316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 07:53:19.699898 4316 server_base.cc:1047] running on GCE node
W20250905 07:53:19.699852 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:19.699820 4321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:19.699821 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:19.700764 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:19.701726 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:19.702890 4316 hybrid_clock.cc:648] HybridClock initialized: now 1757058799702875 us; error 36 us; skew 500 ppm
Sep 05 07:53:19 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058799, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:19.705572 4316 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:19.706769 4316 webserver.cc:480] Webserver started at http://127.4.49.62:46125/ using document root <none> and password file <none>
I20250905 07:53:19.707358 4316 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:19.707414 4316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:19.707623 4316 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:19.709553 4316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "510078f39f584698885ca48f4d64fcaa"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "06b566531c304a1a186af49f6cc9aa1e"
server_key_iv: "9801791f8a7df2242726a5b0201b5ed0"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.710052 4316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "510078f39f584698885ca48f4d64fcaa"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "06b566531c304a1a186af49f6cc9aa1e"
server_key_iv: "9801791f8a7df2242726a5b0201b5ed0"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.713707 4316 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250905 07:53:19.716075 4331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:19.717190 4316 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:19.717341 4316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "510078f39f584698885ca48f4d64fcaa"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "06b566531c304a1a186af49f6cc9aa1e"
server_key_iv: "9801791f8a7df2242726a5b0201b5ed0"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.717471 4316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:19.738230 4316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:19.741223 4316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:19.741432 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:19.748711 4316 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:44457
I20250905 07:53:19.748718 4384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:44457 every 8 connection(s)
I20250905 07:53:19.749825 4316 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:19.751824 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4316
I20250905 07:53:19.751950 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:19.752840 4385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:19.753021 4292 external_mini_cluster.cc:1468] Setting key 2c9f4c79361a60303240deb546e38034
I20250905 07:53:19.760020 4385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa: Bootstrap starting.
I20250905 07:53:19.762254 4385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:19.762964 4385 log.cc:826] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:19.765007 4385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa: No bootstrap required, opened a new log
Sep 05 07:53:19 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:19.767867 4385 raft_consensus.cc:357] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } }
I20250905 07:53:19.768090 4385 raft_consensus.cc:383] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:19.768165 4385 raft_consensus.cc:738] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510078f39f584698885ca48f4d64fcaa, State: Initialized, Role: FOLLOWER
I20250905 07:53:19.768642 4385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } }
I20250905 07:53:19.768808 4385 raft_consensus.cc:397] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:19.768877 4385 raft_consensus.cc:491] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:19.768990 4385 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:19.769971 4385 raft_consensus.cc:513] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } }
I20250905 07:53:19.770378 4385 leader_election.cc:304] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510078f39f584698885ca48f4d64fcaa; no voters:
I20250905 07:53:19.770733 4385 leader_election.cc:290] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:19.770898 4390 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:19.771119 4390 raft_consensus.cc:695] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [term 1 LEADER]: Becoming Leader. State: Replica: 510078f39f584698885ca48f4d64fcaa, State: Running, Role: LEADER
I20250905 07:53:19.771468 4390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } }
I20250905 07:53:19.771723 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:19.755446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.4.49.62:44457)
0905 07:53:19.755973 (+ 527us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:19.755985 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:19.756552 (+ 567us) server_negotiation.cc:408] Connection header received
0905 07:53:19.759038 (+ 2486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:19.759061 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:19.759391 (+ 330us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:19.759699 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:19.761444 (+ 1745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:19.762192 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:19.762888 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:19.763148 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:19.766555 (+ 3407us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:19.766577 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:19.766588 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:19.766630 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:19.768386 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:19.769055 (+ 669us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:19.769061 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:19.769066 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:19.769160 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:19.769481 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:19.769483 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:19.769485 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:19.769839 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:19.769995 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:19.770614 (+ 619us) server_negotiation.cc:300] Negotiation successful
0905 07:53:19.770868 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":134,"threads_started":1}
I20250905 07:53:19.771950 4385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:19.773058 4391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510078f39f584698885ca48f4d64fcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } } }
I20250905 07:53:19.773152 4391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:19.773367 4391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510078f39f584698885ca48f4d64fcaa. Latest consensus state: current_term: 1 leader_uuid: "510078f39f584698885ca48f4d64fcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510078f39f584698885ca48f4d64fcaa" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44457 } } }
I20250905 07:53:19.773449 4391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:19.773778 4395 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:19.776719 4395 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:19.782577 4395 catalog_manager.cc:1349] Generated new cluster ID: aed19a30af4246e7867404a80697919a
I20250905 07:53:19.782660 4395 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:19.791944 4395 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:19.792866 4395 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:19.800813 4395 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 510078f39f584698885ca48f4d64fcaa: Generated new TSK 0
I20250905 07:53:19.801522 4395 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:19.846704 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44457
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:19.954246 4413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:19.954486 4413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:19.954550 4413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:19.957876 4413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:19.957952 4413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:19.958037 4413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:19.962450 4413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:19.963526 4413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:19.964370 4413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:19.970752 4421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:19.970744 4418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:19.970721 4419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:19.971062 4413 server_base.cc:1047] running on GCE node
I20250905 07:53:19.971419 4413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:19.971966 4413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:19.973115 4413 hybrid_clock.cc:648] HybridClock initialized: now 1757058799973089 us; error 37 us; skew 500 ppm
Sep 05 07:53:19 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058799, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:19.975795 4413 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:19.976855 4413 webserver.cc:480] Webserver started at http://127.4.49.1:42045/ using document root <none> and password file <none>
I20250905 07:53:19.977381 4413 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:19.977427 4413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:19.977595 4413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:19.979280 4413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "52d9e41482584eebb9a648900fbc47d8"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "eb20ab9c956052bb56dcffc7aef69d7d"
server_key_iv: "46e01e4ee6fb2d3b28a3873b9a9a968f"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.979750 4413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "52d9e41482584eebb9a648900fbc47d8"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "eb20ab9c956052bb56dcffc7aef69d7d"
server_key_iv: "46e01e4ee6fb2d3b28a3873b9a9a968f"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.983170 4413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:53:19.985466 4428 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:19.986469 4413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:19.986594 4413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "52d9e41482584eebb9a648900fbc47d8"
format_stamp: "Formatted at 2025-09-05 07:53:19 on dist-test-slave-mnfb"
server_key: "eb20ab9c956052bb56dcffc7aef69d7d"
server_key_iv: "46e01e4ee6fb2d3b28a3873b9a9a968f"
server_key_version: "encryptionkey@0"
I20250905 07:53:19.986693 4413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:19.997586 4413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:20.000689 4413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:20.000862 4413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:20.001405 4413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:20.002262 4413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:20.002308 4413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.002355 4413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:20.002383 4413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.010906 4413 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:43961
I20250905 07:53:20.010938 4543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:43961 every 8 connection(s)
I20250905 07:53:20.011894 4413 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:20.012132 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4413
I20250905 07:53:20.012214 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:20.012480 4292 external_mini_cluster.cc:1468] Setting key c10a81b6bf4a78917cf6d5ed84dcb757
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058799, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:20.027701 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.013867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:54217 (local address 127.4.49.62:44457)
0905 07:53:20.013989 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:20.013992 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:20.014788 (+ 796us) server_negotiation.cc:408] Connection header received
0905 07:53:20.015645 (+ 857us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:20.015649 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:20.015705 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:20.015775 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:20.017164 (+ 1389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.017689 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.018312 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.018496 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.023668 (+ 5172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:20.023695 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:20.023697 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:20.023729 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:20.025465 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.026061 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.026066 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.026068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.026134 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.026429 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.026432 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.026434 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.026666 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:20.026774 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:20.027334 (+ 560us) server_negotiation.cc:300] Negotiation successful
0905 07:53:20.027516 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:53:20.028314 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.014186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44457 (local address 127.4.49.1:54217)
0905 07:53:20.014638 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:20.014682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:20.015444 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:20.015954 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:20.015963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:20.016438 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:20.017015 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.017029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.017804 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.017807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:20.018191 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.018198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.018417 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.019599 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:20.019624 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:20.023469 (+ 3845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:20.025624 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.025630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.025643 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.025944 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.026231 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.026233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.026235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.026343 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.026801 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:20.026808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:20.027072 (+ 264us) client_negotiation.cc:769] Sending connection context
0905 07:53:20.027261 (+ 189us) client_negotiation.cc:240] Negotiation successful
0905 07:53:20.027513 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20250905 07:53:20.029582 4544 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44457
I20250905 07:53:20.029881 4544 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:20.030527 4544 heartbeater.cc:507] Master 127.4.49.62:44457 requested a full tablet report, sending...
I20250905 07:53:20.032198 4348 ts_manager.cc:194] Registered new tserver with Master: 52d9e41482584eebb9a648900fbc47d8 (127.4.49.1:43961)
I20250905 07:53:20.033596 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:54217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:20.056293 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44457
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:20.164237 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:20.164479 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:20.164558 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:20.167896 4551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:20.167971 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:20.168112 4551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:20.172585 4551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:20.173864 4551 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:20.174916 4551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:20.182705 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:20.182705 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:20.182837 4551 server_base.cc:1047] running on GCE node
W20250905 07:53:20.182708 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:20.183359 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:20.183930 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:20.185098 4551 hybrid_clock.cc:648] HybridClock initialized: now 1757058800185076 us; error 39 us; skew 500 ppm
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:20.187878 4551 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:20.189033 4551 webserver.cc:480] Webserver started at http://127.4.49.2:34307/ using document root <none> and password file <none>
I20250905 07:53:20.189636 4551 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:20.189694 4551 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:20.189885 4551 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:20.191810 4551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "31a965e80f654ae3a8896817963110ff"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "9d214b573ec8c35cbeab46328f63dc89"
server_key_iv: "c2bdde3cfe056456bf4a289ccb64b9cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.192368 4551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "31a965e80f654ae3a8896817963110ff"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "9d214b573ec8c35cbeab46328f63dc89"
server_key_iv: "c2bdde3cfe056456bf4a289ccb64b9cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.195838 4551 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:20.198403 4566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.199666 4551 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:20.199791 4551 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "31a965e80f654ae3a8896817963110ff"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "9d214b573ec8c35cbeab46328f63dc89"
server_key_iv: "c2bdde3cfe056456bf4a289ccb64b9cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.199893 4551 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:20.211469 4551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:20.214479 4551 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:20.214663 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:20.215198 4551 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:20.216161 4551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:20.216212 4551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.216262 4551 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:20.216287 4551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.225227 4551 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:32777
I20250905 07:53:20.225272 4682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:32777 every 8 connection(s)
I20250905 07:53:20.226178 4551 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:20.232234 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4551
I20250905 07:53:20.232379 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:20.232642 4292 external_mini_cluster.cc:1468] Setting key b70b617d14e2e97694816c18a549f6a3
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:20.239213 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.227964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:49421 (local address 127.4.49.62:44457)
0905 07:53:20.228167 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:20.228172 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:20.228892 (+ 720us) server_negotiation.cc:408] Connection header received
0905 07:53:20.229681 (+ 789us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:20.229685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:20.229736 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:20.229806 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:20.231078 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.231590 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.232266 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.232439 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.235580 (+ 3141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:20.235603 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:20.235607 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:20.235642 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:20.237190 (+ 1548us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.237716 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.237720 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.237722 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.237780 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.238043 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.238046 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.238048 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.238262 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:20.238372 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:20.238891 (+ 519us) server_negotiation.cc:300] Negotiation successful
0905 07:53:20.239029 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250905 07:53:20.239789 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.228293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44457 (local address 127.4.49.2:49421)
0905 07:53:20.228757 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:20.228793 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:20.229499 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:20.229953 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:20.229961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:20.230358 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:20.230950 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.230964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.231689 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.231691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:20.232161 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.232172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.232393 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.233667 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:20.233695 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:20.235382 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:20.237347 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.237353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.237365 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.237609 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.237868 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.237870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.237872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.237962 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.238381 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:20.238387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:20.238643 (+ 256us) client_negotiation.cc:769] Sending connection context
0905 07:53:20.238821 (+ 178us) client_negotiation.cc:240] Negotiation successful
0905 07:53:20.239047 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20250905 07:53:20.240902 4683 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44457
I20250905 07:53:20.241218 4683 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:53:20.241942 4683 heartbeater.cc:507] Master 127.4.49.62:44457 requested a full tablet report, sending...
I20250905 07:53:20.243091 4348 ts_manager.cc:194] Registered new tserver with Master: 31a965e80f654ae3a8896817963110ff (127.4.49.2:32777)
I20250905 07:53:20.243723 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:49421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:20.276563 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44457
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:20.384680 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:20.384934 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:20.384997 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:20.388412 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:20.388489 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:20.388576 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:20.393258 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:20.394389 4690 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:20.395247 4690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:20.401824 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:20.401840 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:20.401818 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:20.402089 4690 server_base.cc:1047] running on GCE node
I20250905 07:53:20.402588 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:20.403175 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:20.404378 4690 hybrid_clock.cc:648] HybridClock initialized: now 1757058800404351 us; error 39 us; skew 500 ppm
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:20.407414 4690 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:20.408668 4690 webserver.cc:480] Webserver started at http://127.4.49.3:40615/ using document root <none> and password file <none>
I20250905 07:53:20.409273 4690 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:20.409330 4690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:20.409531 4690 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:20.411402 4690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "869eb5b036a446748842c99ac7171e5a"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "5c1fd68f696eeb4edefe887fcb93115b"
server_key_iv: "efad3fe7245c5a05568c65a288ead433"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.411885 4690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "869eb5b036a446748842c99ac7171e5a"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "5c1fd68f696eeb4edefe887fcb93115b"
server_key_iv: "efad3fe7245c5a05568c65a288ead433"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.415570 4690 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:20.418098 4705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.419214 4690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:20.419373 4690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "869eb5b036a446748842c99ac7171e5a"
format_stamp: "Formatted at 2025-09-05 07:53:20 on dist-test-slave-mnfb"
server_key: "5c1fd68f696eeb4edefe887fcb93115b"
server_key_iv: "efad3fe7245c5a05568c65a288ead433"
server_key_version: "encryptionkey@0"
I20250905 07:53:20.419478 4690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:20.433837 4690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:20.437095 4690 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:20.437285 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:20.437860 4690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:20.438756 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:20.438805 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.438859 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:20.438884 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:20.448092 4690 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:35775
I20250905 07:53:20.448130 4819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:35775 every 8 connection(s)
I20250905 07:53:20.449149 4690 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:20.451929 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4690
I20250905 07:53:20.452037 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:20.452306 4292 external_mini_cluster.cc:1468] Setting key 7635fca54344c164f4d4a255e1b93b71
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:20.462352 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.450985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:40197 (local address 127.4.49.62:44457)
0905 07:53:20.451165 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:20.451170 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:20.452010 (+ 840us) server_negotiation.cc:408] Connection header received
0905 07:53:20.453067 (+ 1057us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:20.453071 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:20.453130 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:20.453203 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:20.454839 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.455343 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.455955 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.456136 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.459079 (+ 2943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:20.459105 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:20.459107 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:20.459139 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:20.460580 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.461112 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.461116 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.461117 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.461167 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.461393 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.461395 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.461396 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.461557 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:20.461641 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:20.462103 (+ 462us) server_negotiation.cc:300] Negotiation successful
0905 07:53:20.462213 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20250905 07:53:20.463094 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.451323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44457 (local address 127.4.49.3:40197)
0905 07:53:20.451825 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:20.451878 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:20.452867 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:20.453395 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:20.453405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:20.453927 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:20.454694 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.454711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.455452 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.455455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:20.455862 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:20.455869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.456040 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.457189 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:20.457216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:20.458885 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:20.460738 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.460744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.460756 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.461014 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.461232 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:20.461234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:20.461236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:20.461325 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:20.461630 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:20.461636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:20.461891 (+ 255us) client_negotiation.cc:769] Sending connection context
0905 07:53:20.462076 (+ 185us) client_negotiation.cc:240] Negotiation successful
0905 07:53:20.462321 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
I20250905 07:53:20.464252 4820 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44457
I20250905 07:53:20.464534 4820 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:20.465137 4820 heartbeater.cc:507] Master 127.4.49.62:44457 requested a full tablet report, sending...
I20250905 07:53:20.466239 4348 ts_manager.cc:194] Registered new tserver with Master: 869eb5b036a446748842c99ac7171e5a (127.4.49.3:35775)
I20250905 07:53:20.466722 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:53:20.466884 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:40197
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:20.490844 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.482304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.4.49.1:43961)
0905 07:53:20.482654 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:20.482661 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:20.482685 (+ 24us) server_negotiation.cc:408] Connection header received
0905 07:53:20.482737 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:20.482742 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:20.482911 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:20.483051 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:20.483904 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.484677 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.485398 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.485661 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.487709 (+ 2048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:20.487741 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:20.487754 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:20.487787 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:20.489411 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.489800 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.489805 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.489807 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.489858 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.490024 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.490026 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.490028 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.490220 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:20.490366 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:20.490514 (+ 148us) server_negotiation.cc:300] Negotiation successful
0905 07:53:20.490682 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
W20250905 07:53:20.491725 4456 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:20.515246 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:20.508252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.4.49.1:43961)
0905 07:53:20.508411 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:20.508415 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:20.508481 (+ 66us) server_negotiation.cc:408] Connection header received
0905 07:53:20.508613 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:20.508616 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:20.508670 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:20.508733 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:20.509481 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.509949 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:20.510535 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:20.510666 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:20.512794 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:20.512816 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:20.512819 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:20.512847 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:20.514048 (+ 1201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.514408 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.514411 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.514412 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.514450 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:20.514595 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:20.514597 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:20.514598 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:20.514734 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:20.514843 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:20.514983 (+ 140us) server_negotiation.cc:300] Negotiation successful
0905 07:53:20.515081 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250905 07:53:20.516212 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4413
I20250905 07:53:20.521957 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4551
I20250905 07:53:20.528723 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4690
I20250905 07:53:20.534196 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4316
2025-09-05T07:53:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2948 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4834](info): set up 2 sockets
Sep 05 07:53:20 dist-test-slave-mnfb krb5kdc[4834](info): commencing operation
krb5kdc: starting...
W20250905 07:53:22.374593 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:22 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:22.506729 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:35839
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:22.613478 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:22.613735 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:22.613812 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:22.617214 4851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:22.617295 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:22.617357 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:22.617393 4851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:22.617425 4851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:22.622025 4851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:22.623162 4851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:22.624117 4851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:22.629735 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:22.629727 4856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:22.629812 4851 server_base.cc:1047] running on GCE node
W20250905 07:53:22.629734 4857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:22.630481 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:22.631393 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:22.632544 4851 hybrid_clock.cc:648] HybridClock initialized: now 1757058802632525 us; error 32 us; skew 500 ppm
Sep 05 07:53:22 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:22.635231 4851 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:22.636469 4851 webserver.cc:480] Webserver started at http://127.4.49.62:33303/ using document root <none> and password file <none>
I20250905 07:53:22.637095 4851 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:22.637152 4851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:22.637375 4851 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:22.639165 4851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "a0d44bfb9a9b6f924347dae3469eea0e"
server_key_iv: "edfdb7eaf28249f82b7a83b7da9ab51b"
server_key_version: "encryptionkey@0"
I20250905 07:53:22.639659 4851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "a0d44bfb9a9b6f924347dae3469eea0e"
server_key_iv: "edfdb7eaf28249f82b7a83b7da9ab51b"
server_key_version: "encryptionkey@0"
I20250905 07:53:22.643185 4851 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:22.645594 4866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:22.646687 4851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:22.646798 4851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "a0d44bfb9a9b6f924347dae3469eea0e"
server_key_iv: "edfdb7eaf28249f82b7a83b7da9ab51b"
server_key_version: "encryptionkey@0"
I20250905 07:53:22.646898 4851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:22.667881 4851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:22.675880 4851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:22.676102 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:22.683393 4851 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:35839
I20250905 07:53:22.683393 4920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:35839 every 8 connection(s)
I20250905 07:53:22.684523 4851 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:22.687377 4921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:22.692431 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4851
I20250905 07:53:22.692533 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:22.692735 4292 external_mini_cluster.cc:1468] Setting key 8afe61d1b0b145b8696df0c96cb4c024
I20250905 07:53:22.693270 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: Bootstrap starting.
I20250905 07:53:22.695956 4921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:22.696830 4921 log.cc:826] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:22.698975 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: No bootstrap required, opened a new log
Sep 05 07:53:22 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:22.702011 4921 raft_consensus.cc:357] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } }
I20250905 07:53:22.702267 4921 raft_consensus.cc:383] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:22.702343 4921 raft_consensus.cc:738] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee25cbd23c3f43b1b88a1dccbe9fb90f, State: Initialized, Role: FOLLOWER
I20250905 07:53:22.702873 4921 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } }
I20250905 07:53:22.703028 4921 raft_consensus.cc:397] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:22.703096 4921 raft_consensus.cc:491] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:22.703213 4921 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:22.704347 4921 raft_consensus.cc:513] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } }
I20250905 07:53:22.704761 4921 leader_election.cc:304] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee25cbd23c3f43b1b88a1dccbe9fb90f; no voters:
I20250905 07:53:22.705134 4921 leader_election.cc:290] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:22.705243 4926 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:22.705446 4926 raft_consensus.cc:695] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [term 1 LEADER]: Becoming Leader. State: Replica: ee25cbd23c3f43b1b88a1dccbe9fb90f, State: Running, Role: LEADER
I20250905 07:53:22.705777 4926 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } }
I20250905 07:53:22.705884 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:22.694023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.4.49.62:35839)
0905 07:53:22.694537 (+ 514us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:22.694550 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:22.694603 (+ 53us) server_negotiation.cc:408] Connection header received
0905 07:53:22.695334 (+ 731us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:22.695362 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:22.695750 (+ 388us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:22.696164 (+ 414us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:22.696904 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:22.697879 (+ 975us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:22.698549 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:22.698815 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:22.700991 (+ 2176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:22.701017 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:22.701034 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:22.701078 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:22.703037 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:22.703488 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:22.703494 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:22.703500 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:22.703602 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:22.703792 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:22.703796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:22.703798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:22.704296 (+ 498us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:22.704518 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:22.704768 (+ 250us) server_negotiation.cc:300] Negotiation successful
0905 07:53:22.705068 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":136,"threads_started":1}
I20250905 07:53:22.706331 4921 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:22.707334 4927 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee25cbd23c3f43b1b88a1dccbe9fb90f. Latest consensus state: current_term: 1 leader_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } } }
I20250905 07:53:22.707453 4927 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:22.707979 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee25cbd23c3f43b1b88a1dccbe9fb90f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35839 } } }
I20250905 07:53:22.708782 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f [sys.catalog]: This master's current role is: LEADER
W20250905 07:53:22.710834 4941 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:22.710896 4941 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:22.711001 4935 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:22.711915 4935 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:22.717760 4935 catalog_manager.cc:1349] Generated new cluster ID: 22e61f6737684cfaa6fe40651678a943
I20250905 07:53:22.717846 4935 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:22.763406 4935 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:22.764814 4935 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:22.796566 4935 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ee25cbd23c3f43b1b88a1dccbe9fb90f: Generated new TSK 0
I20250905 07:53:22.797350 4935 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:22.851322 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35839
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:22.972919 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:22.973165 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:22.973232 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:22.976727 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:22.976812 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:22.976914 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:22.981544 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:22.982756 4950 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:22.983711 4950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 07:53:22.990865 4950 server_base.cc:1047] running on GCE node
W20250905 07:53:22.990816 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:22.990816 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:22.990818 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:22.991638 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:22.992292 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:22.993487 4950 hybrid_clock.cc:648] HybridClock initialized: now 1757058802993475 us; error 33 us; skew 500 ppm
Sep 05 07:53:22 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:22.996486 4950 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:22.997756 4950 webserver.cc:480] Webserver started at http://127.4.49.1:46657/ using document root <none> and password file <none>
I20250905 07:53:22.998337 4950 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:22.998400 4950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:22.998587 4950 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:23.000494 4950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "1cabdc0b0735d003e516e14b976e9bb9"
server_key_iv: "46baaa51d80ebecce5c0d1a5c7601332"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.001012 4950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "1cabdc0b0735d003e516e14b976e9bb9"
server_key_iv: "46baaa51d80ebecce5c0d1a5c7601332"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.004703 4950 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:23.007175 4965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.008503 4950 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:23.008639 4950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d"
format_stamp: "Formatted at 2025-09-05 07:53:22 on dist-test-slave-mnfb"
server_key: "1cabdc0b0735d003e516e14b976e9bb9"
server_key_iv: "46baaa51d80ebecce5c0d1a5c7601332"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.008747 4950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:23.030066 4950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:23.033226 4950 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:23.033435 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:23.034020 4950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:23.034942 4950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:23.034996 4950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.035051 4950 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:23.035076 4950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.045022 4950 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:39049
I20250905 07:53:23.045035 5080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:39049 every 8 connection(s)
I20250905 07:53:23.046074 4950 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:23.048044 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 4950
I20250905 07:53:23.048174 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:23.048410 4292 external_mini_cluster.cc:1468] Setting key 3681f6212d1ffa29cf3ccb61bd44b193
Sep 05 07:53:23 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:53:23.060251 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.047944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:53183 (local address 127.4.49.62:35839)
0905 07:53:23.048167 (+ 223us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.048172 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.049415 (+ 1243us) server_negotiation.cc:408] Connection header received
0905 07:53:23.050327 (+ 912us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.050330 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.050395 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.050469 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:23.052526 (+ 2057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.053056 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.053785 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.053970 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.056359 (+ 2389us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:23.056389 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:23.056393 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:23.056434 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:23.057893 (+ 1459us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.058452 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.058456 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.058457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.058512 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.058936 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.058940 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.058942 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.059144 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:23.059248 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.059905 (+ 657us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.060078 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20250905 07:53:23.060854 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.048248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35839 (local address 127.4.49.1:53183)
0905 07:53:23.049280 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.049319 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.050111 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.050684 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.050693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.051178 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:23.052246 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.052273 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.053211 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.053214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.053658 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.053666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.053887 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.054602 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:23.054630 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:23.056176 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:23.058027 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.058032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.058045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.058304 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.058645 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.058648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.058650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.058759 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.059259 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:23.059267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:23.059582 (+ 315us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.059785 (+ 203us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.060033 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":117,"threads_started":1}
I20250905 07:53:23.062239 5081 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35839
I20250905 07:53:23.062568 5081 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:23.063203 5081 heartbeater.cc:507] Master 127.4.49.62:35839 requested a full tablet report, sending...
I20250905 07:53:23.065150 4884 ts_manager.cc:194] Registered new tserver with Master: dbe8cbd5fa3f4a4e8964fc753d0fd29d (127.4.49.1:39049)
I20250905 07:53:23.066895 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:23.099195 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35839
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:23.212013 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:23.212355 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:23.212425 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:23.216008 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:23.216140 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:23.216235 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:23.220971 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:23.222102 5088 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:23.222987 5088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:23.230134 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:23.230134 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:23.230221 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:23.230500 5088 server_base.cc:1047] running on GCE node
I20250905 07:53:23.230938 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:23.231564 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:23.232769 5088 hybrid_clock.cc:648] HybridClock initialized: now 1757058803232713 us; error 73 us; skew 500 ppm
Sep 05 07:53:23 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058803, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:23.236179 5088 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:23.237584 5088 webserver.cc:480] Webserver started at http://127.4.49.2:38363/ using document root <none> and password file <none>
I20250905 07:53:23.238183 5088 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:23.238240 5088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:23.238443 5088 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:23.240393 5088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "6966567bc0ba48eaa3c453e0dcd88050"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "e6aa6e1aa5a1825504caa8cfdb123726"
server_key_iv: "166abfd057cae6774a95c876f3842924"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.240916 5088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "6966567bc0ba48eaa3c453e0dcd88050"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "e6aa6e1aa5a1825504caa8cfdb123726"
server_key_iv: "166abfd057cae6774a95c876f3842924"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.244649 5088 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:23.247231 5103 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.248521 5088 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:23.248670 5088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "6966567bc0ba48eaa3c453e0dcd88050"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "e6aa6e1aa5a1825504caa8cfdb123726"
server_key_iv: "166abfd057cae6774a95c876f3842924"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.248780 5088 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:23.261374 5088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:23.264880 5088 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:23.265082 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:23.265671 5088 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:23.266705 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:23.266770 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.266831 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:23.266860 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.276021 5088 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:46315
I20250905 07:53:23.276073 5218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:46315 every 8 connection(s)
I20250905 07:53:23.277099 5088 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:23.285530 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5088
I20250905 07:53:23.285634 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:23.285843 4292 external_mini_cluster.cc:1468] Setting key cc8044308f8ba87f2ee082e5f1381d0c
Sep 05 07:53:23 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058803, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:23.290427 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.279011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:43579 (local address 127.4.49.62:35839)
0905 07:53:23.279164 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.279169 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.279948 (+ 779us) server_negotiation.cc:408] Connection header received
0905 07:53:23.280834 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.280837 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.280892 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.280955 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:23.282320 (+ 1365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.282851 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.283556 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.283694 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.286737 (+ 3043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:23.286757 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:23.286759 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:23.286788 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:23.288294 (+ 1506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.288941 (+ 647us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.288944 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.288946 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.289014 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.289298 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.289301 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.289303 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.289482 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:23.289584 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.290109 (+ 525us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.290254 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250905 07:53:23.291224 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.279314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35839 (local address 127.4.49.2:43579)
0905 07:53:23.279798 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.279840 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.280617 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.281120 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.281130 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.281549 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:23.282153 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.282168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.283012 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.283015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.283450 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.283459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.283681 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.284845 (+ 1164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:23.284877 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:23.286555 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:23.288449 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.288457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.288473 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.288824 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.289104 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.289106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.289108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.289199 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.289595 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:23.289603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:23.289855 (+ 252us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.290055 (+ 200us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.290368 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
I20250905 07:53:23.292511 5219 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35839
I20250905 07:53:23.292819 5219 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:23.293567 5219 heartbeater.cc:507] Master 127.4.49.62:35839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:53:23.294836 4884 ts_manager.cc:194] Registered new tserver with Master: 6966567bc0ba48eaa3c453e0dcd88050 (127.4.49.2:46315)
I20250905 07:53:23.295428 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:43579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:23.330979 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35839
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:23.445626 5226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:23.445870 5226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:23.445936 5226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:23.449347 5226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:23.449437 5226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:23.449532 5226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:23.454005 5226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:23.455116 5226 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:23.455997 5226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 07:53:23.462939 5226 server_base.cc:1047] running on GCE node
W20250905 07:53:23.462888 5232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:23.462888 5234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:23.462888 5231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:23.463658 5226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:23.464300 5226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:23.465451 5226 hybrid_clock.cc:648] HybridClock initialized: now 1757058803465440 us; error 28 us; skew 500 ppm
Sep 05 07:53:23 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058803, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:23.468266 5226 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:23.469357 5226 webserver.cc:480] Webserver started at http://127.4.49.3:35195/ using document root <none> and password file <none>
I20250905 07:53:23.469898 5226 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:23.469947 5226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:23.470122 5226 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:23.471889 5226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "38917ff4417d4d1fa041100ac6e47fa0"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "6c0261f6a89c70d0aad522019da49358"
server_key_iv: "2c2d64b743ad260ce8acd3b962b061be"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.472375 5226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "38917ff4417d4d1fa041100ac6e47fa0"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "6c0261f6a89c70d0aad522019da49358"
server_key_iv: "2c2d64b743ad260ce8acd3b962b061be"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.475868 5226 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:23.478267 5242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.479362 5226 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:23.479494 5226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "38917ff4417d4d1fa041100ac6e47fa0"
format_stamp: "Formatted at 2025-09-05 07:53:23 on dist-test-slave-mnfb"
server_key: "6c0261f6a89c70d0aad522019da49358"
server_key_iv: "2c2d64b743ad260ce8acd3b962b061be"
server_key_version: "encryptionkey@0"
I20250905 07:53:23.479593 5226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:23.492796 5226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:23.495770 5226 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:23.495954 5226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:23.496559 5226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:23.497452 5226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:23.497500 5226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.497551 5226 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:23.497577 5226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:23.506198 5226 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:45483
I20250905 07:53:23.506218 5358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:45483 every 8 connection(s)
I20250905 07:53:23.507202 5226 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:23.507558 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5226
I20250905 07:53:23.507683 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:23.507926 4292 external_mini_cluster.cc:1468] Setting key 46284bdc82b65afa80ff082bb78eb972
Sep 05 07:53:23 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058803, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:23.519179 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.509109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35945 (local address 127.4.49.62:35839)
0905 07:53:23.509239 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.509243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.510101 (+ 858us) server_negotiation.cc:408] Connection header received
0905 07:53:23.510940 (+ 839us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.510943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.510993 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.511057 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:23.512453 (+ 1396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.512979 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.513578 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.513750 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.516429 (+ 2679us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:23.516452 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:23.516454 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:23.516483 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:23.517659 (+ 1176us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.518082 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.518084 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.518086 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.518121 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.518313 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.518315 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.518316 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.518458 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:23.518523 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.518955 (+ 432us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.519053 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:53:23.519856 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.509485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35839 (local address 127.4.49.3:35945)
0905 07:53:23.509950 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.509997 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.510762 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.511199 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.511207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.511639 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:23.512307 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.512326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.513074 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.513077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.513494 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.513501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.513684 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.514739 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:23.514765 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:23.516262 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:23.517764 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.517768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.517780 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.518008 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.518177 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:23.518179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:23.518180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:23.518257 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:23.518523 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:23.518528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:23.518758 (+ 230us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.518928 (+ 170us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.519152 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
I20250905 07:53:23.521226 5359 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35839
I20250905 07:53:23.521492 5359 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:23.522081 5359 heartbeater.cc:507] Master 127.4.49.62:35839 requested a full tablet report, sending...
I20250905 07:53:23.523005 4884 ts_manager.cc:194] Registered new tserver with Master: 38917ff4417d4d1fa041100ac6e47fa0 (127.4.49.3:45483)
I20250905 07:53:23.523550 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:35945
I20250905 07:53:23.533676 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:53:23.543040 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.536640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.4.49.62:35839)
0905 07:53:23.536781 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.536786 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.536907 (+ 121us) server_negotiation.cc:408] Connection header received
0905 07:53:23.537023 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.537026 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.537077 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.537143 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:23.538003 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.538509 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.539139 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.539263 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.540405 (+ 1142us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:23.540424 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:23.540427 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:23.540455 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:23.541597 (+ 1142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.542013 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.542016 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.542017 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.542065 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:23.542294 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:23.542296 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:23.542297 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:23.542491 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:23.542591 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.542771 (+ 180us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.542871 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:53:23.547256 4884 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:23.549724 4884 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:23.562067 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.557913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:39049 (local address 127.0.0.1:34122)
0905 07:53:23.558365 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.558410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.558569 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.559295 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.559301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.559329 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:23.559653 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.559663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.560867 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.560870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.561543 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.561553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.561659 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.561724 (+ 65us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.561815 (+ 91us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.561914 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":6784,"thread_start_us":84,"threads_started":1}
I20250905 07:53:23.563122 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.558882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.4.49.2:46315)
0905 07:53:23.559177 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.559182 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.559203 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:53:23.559288 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.559293 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.559453 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.559592 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:23.559964 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.560831 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.562145 (+ 1314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.562696 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.562822 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.562888 (+ 66us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.562990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20250905 07:53:23.563179 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.558066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.4.49.1:39049)
0905 07:53:23.558818 (+ 752us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.558825 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.558846 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:53:23.558928 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.558934 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.559125 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.559282 (+ 157us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:23.559786 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.560761 (+ 975us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.562454 (+ 1693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.562808 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.562910 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.562969 (+ 59us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.563054 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":145,"threads_started":1}
I20250905 07:53:23.562254 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.558044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:46315 (local address 127.0.0.1:45442)
0905 07:53:23.558438 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.558449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.558569 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.559596 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.559599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.559618 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:23.559856 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.559862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.560925 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.560929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.561816 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.561823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.561912 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.561930 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.561973 (+ 43us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.562026 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"spinlock_wait_cycles":6528,"thread_start_us":89,"threads_started":1}
I20250905 07:53:23.563112 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.558591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:45483 (local address 127.0.0.1:53194)
0905 07:53:23.559640 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.559653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.559728 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.560116 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.560119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.560138 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:23.560367 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.560373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.561698 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.561701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.562770 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.562780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.562888 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.562904 (+ 16us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.562947 (+ 43us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.562996 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":99,"threads_started":1}
I20250905 07:53:23.563860 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.558759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.4.49.3:45483)
0905 07:53:23.559156 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.559162 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.559738 (+ 576us) server_negotiation.cc:408] Connection header received
0905 07:53:23.559825 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.559830 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.559976 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.560130 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:23.560484 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.561560 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.562933 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.563446 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.563546 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.563615 (+ 69us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.563721 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":140,"threads_started":1}
I20250905 07:53:23.566220 5013 tablet_service.cc:1468] Processing CreateTablet for tablet 44ef14e9a3a7473084a9402958e43710 (DEFAULT_TABLE table=test-table [id=327cfa6d109c47f9bebd2502e34b2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:23.566358 5151 tablet_service.cc:1468] Processing CreateTablet for tablet 44ef14e9a3a7473084a9402958e43710 (DEFAULT_TABLE table=test-table [id=327cfa6d109c47f9bebd2502e34b2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:23.566358 5291 tablet_service.cc:1468] Processing CreateTablet for tablet 44ef14e9a3a7473084a9402958e43710 (DEFAULT_TABLE table=test-table [id=327cfa6d109c47f9bebd2502e34b2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:23.567147 5013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef14e9a3a7473084a9402958e43710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:23.567261 5291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef14e9a3a7473084a9402958e43710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:23.567261 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef14e9a3a7473084a9402958e43710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:23.573324 5376 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Bootstrap starting.
I20250905 07:53:23.574299 5378 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Bootstrap starting.
I20250905 07:53:23.574824 5377 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Bootstrap starting.
I20250905 07:53:23.575624 5376 tablet_bootstrap.cc:654] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:23.576435 5376 log.cc:826] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:23.576577 5378 tablet_bootstrap.cc:654] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:23.576731 5377 tablet_bootstrap.cc:654] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:23.577503 5378 log.cc:826] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:23.577503 5377 log.cc:826] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:23.578519 5376 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: No bootstrap required, opened a new log
I20250905 07:53:23.578737 5376 ts_tablet_manager.cc:1397] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250905 07:53:23.579401 5377 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: No bootstrap required, opened a new log
I20250905 07:53:23.579397 5378 tablet_bootstrap.cc:492] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: No bootstrap required, opened a new log
I20250905 07:53:23.579591 5377 ts_tablet_manager.cc:1397] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:53:23.579591 5378 ts_tablet_manager.cc:1397] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:53:23.581270 5376 raft_consensus.cc:357] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.581506 5376 raft_consensus.cc:383] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:23.581593 5376 raft_consensus.cc:738] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe8cbd5fa3f4a4e8964fc753d0fd29d, State: Initialized, Role: FOLLOWER
I20250905 07:53:23.582067 5376 consensus_queue.cc:260] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.582792 5081 heartbeater.cc:499] Master 127.4.49.62:35839 was elected leader, sending a full tablet report...
I20250905 07:53:23.583025 5376 ts_tablet_manager.cc:1428] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 07:53:23.582964 5377 raft_consensus.cc:357] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.583204 5377 raft_consensus.cc:383] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:23.583137 5378 raft_consensus.cc:357] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.583256 5377 raft_consensus.cc:738] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38917ff4417d4d1fa041100ac6e47fa0, State: Initialized, Role: FOLLOWER
I20250905 07:53:23.583307 5378 raft_consensus.cc:383] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:23.583348 5378 raft_consensus.cc:738] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6966567bc0ba48eaa3c453e0dcd88050, State: Initialized, Role: FOLLOWER
I20250905 07:53:23.583813 5378 consensus_queue.cc:260] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.583813 5377 consensus_queue.cc:260] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.584861 5377 ts_tablet_manager.cc:1428] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250905 07:53:23.585233 5359 heartbeater.cc:499] Master 127.4.49.62:35839 was elected leader, sending a full tablet report...
I20250905 07:53:23.585724 5219 heartbeater.cc:499] Master 127.4.49.62:35839 was elected leader, sending a full tablet report...
I20250905 07:53:23.586927 5378 ts_tablet_manager.cc:1428] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20250905 07:53:23.686545 4916 messenger.cc:191] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.4.49.62:35839
W20250905 07:53:23.758387 5360 tablet.cc:2378] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:53:23.778290 5220 tablet.cc:2378] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:53:23.797343 5082 tablet.cc:2378] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:23.968820 5382 raft_consensus.cc:491] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:23.969029 5382 raft_consensus.cc:513] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.970118 5382 leader_election.cc:290] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6966567bc0ba48eaa3c453e0dcd88050 (127.4.49.2:46315), 38917ff4417d4d1fa041100ac6e47fa0 (127.4.49.3:45483)
I20250905 07:53:23.973665 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.970766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:45483 (local address 127.4.49.1:49461)
0905 07:53:23.971086 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.971100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.971198 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.971462 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.971464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.971497 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:23.971747 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.971753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.972596 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.972600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.973199 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.973208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.973311 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.973336 (+ 25us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.973395 (+ 59us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.973460 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20250905 07:53:23.973692 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.970371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:46315 (local address 127.4.49.1:35327)
0905 07:53:23.970722 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:23.970738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:23.970914 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:23.971265 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:23.971268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:23.971292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:23.971863 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.971880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.972647 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.972651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:23.973239 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:23.973247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.973377 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.973397 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:53:23.973447 (+ 50us) client_negotiation.cc:240] Negotiation successful
0905 07:53:23.973507 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20250905 07:53:23.974165 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.970507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:35327 (local address 127.4.49.2:46315)
0905 07:53:23.970676 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.970680 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.970826 (+ 146us) server_negotiation.cc:408] Connection header received
0905 07:53:23.971087 (+ 261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.971091 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.971156 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.971231 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:23.972004 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.972526 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.973374 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.973886 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.973932 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.973987 (+ 55us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.974050 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250905 07:53:23.974404 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:23.970895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:49461 (local address 127.4.49.3:45483)
0905 07:53:23.971073 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:23.971077 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:23.971185 (+ 108us) server_negotiation.cc:408] Connection header received
0905 07:53:23.971291 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:23.971294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:23.971346 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:23.971443 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:23.971882 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.972487 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:23.973331 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:23.974136 (+ 805us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:23.974170 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:23.974217 (+ 47us) server_negotiation.cc:300] Negotiation successful
0905 07:53:23.974282 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250905 07:53:23.974954 5172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44ef14e9a3a7473084a9402958e43710" candidate_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6966567bc0ba48eaa3c453e0dcd88050" is_pre_election: true
I20250905 07:53:23.974982 5312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44ef14e9a3a7473084a9402958e43710" candidate_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38917ff4417d4d1fa041100ac6e47fa0" is_pre_election: true
I20250905 07:53:23.975237 5312 raft_consensus.cc:2466] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbe8cbd5fa3f4a4e8964fc753d0fd29d in term 0.
I20250905 07:53:23.975237 5172 raft_consensus.cc:2466] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbe8cbd5fa3f4a4e8964fc753d0fd29d in term 0.
I20250905 07:53:23.975720 4968 leader_election.cc:304] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6966567bc0ba48eaa3c453e0dcd88050, dbe8cbd5fa3f4a4e8964fc753d0fd29d; no voters:
I20250905 07:53:23.975992 5382 raft_consensus.cc:2802] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:23.976094 5382 raft_consensus.cc:491] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:23.976146 5382 raft_consensus.cc:3058] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:23.977128 5382 raft_consensus.cc:513] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.977525 5382 leader_election.cc:290] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [CANDIDATE]: Term 1 election: Requested vote from peers 6966567bc0ba48eaa3c453e0dcd88050 (127.4.49.2:46315), 38917ff4417d4d1fa041100ac6e47fa0 (127.4.49.3:45483)
I20250905 07:53:23.977995 5172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44ef14e9a3a7473084a9402958e43710" candidate_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6966567bc0ba48eaa3c453e0dcd88050"
I20250905 07:53:23.977999 5312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44ef14e9a3a7473084a9402958e43710" candidate_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38917ff4417d4d1fa041100ac6e47fa0"
I20250905 07:53:23.978137 5172 raft_consensus.cc:3058] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:23.978137 5312 raft_consensus.cc:3058] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:23.979365 5172 raft_consensus.cc:2466] T 44ef14e9a3a7473084a9402958e43710 P 6966567bc0ba48eaa3c453e0dcd88050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbe8cbd5fa3f4a4e8964fc753d0fd29d in term 1.
I20250905 07:53:23.979365 5312 raft_consensus.cc:2466] T 44ef14e9a3a7473084a9402958e43710 P 38917ff4417d4d1fa041100ac6e47fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbe8cbd5fa3f4a4e8964fc753d0fd29d in term 1.
I20250905 07:53:23.979738 4968 leader_election.cc:304] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38917ff4417d4d1fa041100ac6e47fa0, dbe8cbd5fa3f4a4e8964fc753d0fd29d; no voters:
I20250905 07:53:23.979920 5382 raft_consensus.cc:2802] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:23.980160 5382 raft_consensus.cc:695] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [term 1 LEADER]: Becoming Leader. State: Replica: dbe8cbd5fa3f4a4e8964fc753d0fd29d, State: Running, Role: LEADER
I20250905 07:53:23.980492 5382 consensus_queue.cc:237] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } }
I20250905 07:53:23.983963 4884 catalog_manager.cc:5582] T 44ef14e9a3a7473084a9402958e43710 P dbe8cbd5fa3f4a4e8964fc753d0fd29d reported cstate change: term changed from 0 to 1, leader changed from <none> to dbe8cbd5fa3f4a4e8964fc753d0fd29d (127.4.49.1). New cstate: current_term: 1 leader_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe8cbd5fa3f4a4e8964fc753d0fd29d" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6966567bc0ba48eaa3c453e0dcd88050" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 46315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38917ff4417d4d1fa041100ac6e47fa0" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45483 } health_report { overall_health: UNKNOWN } } }
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:24.020830 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:24.012335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.4.49.1:39049)
0905 07:53:24.012570 (+ 235us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:24.012575 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:24.012671 (+ 96us) server_negotiation.cc:408] Connection header received
0905 07:53:24.012834 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:24.012838 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:24.012892 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:24.012966 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:24.013866 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.014350 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:24.015004 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.015198 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:24.017524 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:24.017560 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:24.017573 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:24.017609 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:24.019192 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.019627 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.019632 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.019633 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.019695 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.019864 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.019866 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.019867 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.020139 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:24.020361 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:24.020573 (+ 212us) server_negotiation.cc:300] Negotiation successful
0905 07:53:24.020692 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:24.046279 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:24.038232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.4.49.1:39049)
0905 07:53:24.038379 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:24.038384 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:24.038493 (+ 109us) server_negotiation.cc:408] Connection header received
0905 07:53:24.038643 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:24.038647 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:24.038695 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:24.038764 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:24.039587 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.040134 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:24.040798 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.040986 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:24.043170 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:24.043198 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:24.043200 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:24.043236 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:24.044724 (+ 1488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.045157 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.045160 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.045161 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.045211 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.045431 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.045434 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.045436 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.045586 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:24.045720 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:24.045987 (+ 267us) server_negotiation.cc:300] Negotiation successful
0905 07:53:24.046127 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:24.072939 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:24.065369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.4.49.1:39049)
0905 07:53:24.065523 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:24.065528 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:24.065642 (+ 114us) server_negotiation.cc:408] Connection header received
0905 07:53:24.065760 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:24.065762 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:24.065811 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:24.065882 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:24.066687 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.067161 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:24.067825 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:24.067977 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:24.070172 (+ 2195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:24.070198 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:24.070200 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:24.070237 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:24.071476 (+ 1239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.071865 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.071867 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.071868 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.071908 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:24.072098 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:24.072101 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:24.072103 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:24.072251 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:24.072354 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:24.072621 (+ 267us) server_negotiation.cc:300] Negotiation successful
0905 07:53:24.072761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20250905 07:53:24.074256 4987 tablet_service.cc:2912] Rejecting scan request for tablet 44ef14e9a3a7473084a9402958e43710: Uninitialized: clean time has not yet been initialized
I20250905 07:53:24.075927 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4950
I20250905 07:53:24.081688 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5088
I20250905 07:53:24.087019 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5226
I20250905 07:53:24.092465 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 4851
2025-09-05T07:53:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3558 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[5401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[5401](info): set up 2 sockets
Sep 05 07:53:24 dist-test-slave-mnfb krb5kdc[5401](info): commencing operation
krb5kdc: starting...
W20250905 07:53:25.942466 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.828s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:26.071878 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43965
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:26.180001 5417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:26.180299 5417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:26.180392 5417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:26.183771 5417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:26.183851 5417 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250905 07:53:26.183877 5417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:26.183905 5417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:26.183936 5417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:26.183964 5417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:26.188602 5417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:43965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:26.189767 5417 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:26.190680 5417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:26.196499 5423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.196503 5422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.196503 5425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:26.197033 5417 server_base.cc:1047] running on GCE node
I20250905 07:53:26.197562 5417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:26.198473 5417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:26.199625 5417 hybrid_clock.cc:648] HybridClock initialized: now 1757058806199608 us; error 35 us; skew 500 ppm
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:26.202514 5417 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:26.203677 5417 webserver.cc:480] Webserver started at http://127.4.49.62:35313/ using document root <none> and password file <none>
I20250905 07:53:26.204299 5417 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:26.204365 5417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:26.204557 5417 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:26.206435 5417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "a861480e5a5d44f296ad166b772d4d38"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "124b4614f67caa1e3fd62cfc489656f5"
server_key_iv: "390ce88dbbadf1eefcdfbb55a9a77bb6"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.206976 5417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "a861480e5a5d44f296ad166b772d4d38"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "124b4614f67caa1e3fd62cfc489656f5"
server_key_iv: "390ce88dbbadf1eefcdfbb55a9a77bb6"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.211181 5417 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:26.213845 5432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.215026 5417 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:26.215184 5417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "a861480e5a5d44f296ad166b772d4d38"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "124b4614f67caa1e3fd62cfc489656f5"
server_key_iv: "390ce88dbbadf1eefcdfbb55a9a77bb6"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.215297 5417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:26.230937 5417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:26.234045 5417 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:26.234229 5417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:26.242774 5417 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:43965
I20250905 07:53:26.242805 5496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:43965 every 8 connection(s)
I20250905 07:53:26.243858 5417 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:26.246850 5497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:26.247720 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5417
I20250905 07:53:26.247828 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:26.248106 4292 external_mini_cluster.cc:1468] Setting key 38616c3edc56803415fc06d662bc7cdf
I20250905 07:53:26.253280 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Bootstrap starting.
I20250905 07:53:26.256247 5497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Neither blocks nor log segments found. Creating new log.
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:26.257067 5497 log.cc:826] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:26.259256 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: No bootstrap required, opened a new log
I20250905 07:53:26.261961 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.249780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.4.49.62:43965)
0905 07:53:26.250242 (+ 462us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.250252 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.250291 (+ 39us) server_negotiation.cc:408] Connection header received
0905 07:53:26.250879 (+ 588us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.250898 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.251227 (+ 329us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.251548 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.252709 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.253577 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.254473 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.254707 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.257339 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.257363 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.257378 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.257424 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.259263 (+ 1839us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.259758 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.259764 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.259769 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.259855 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.260130 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.260133 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.260135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.260515 (+ 380us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.260715 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.261024 (+ 309us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.261296 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":153,"threads_started":1}
I20250905 07:53:26.262308 5497 raft_consensus.cc:357] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } }
I20250905 07:53:26.262543 5497 raft_consensus.cc:383] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:26.262606 5497 raft_consensus.cc:738] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a861480e5a5d44f296ad166b772d4d38, State: Initialized, Role: FOLLOWER
I20250905 07:53:26.263060 5497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } }
I20250905 07:53:26.263195 5497 raft_consensus.cc:397] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:26.263254 5497 raft_consensus.cc:491] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:26.263371 5497 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:26.264410 5497 raft_consensus.cc:513] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } }
I20250905 07:53:26.264783 5497 leader_election.cc:304] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a861480e5a5d44f296ad166b772d4d38; no voters:
I20250905 07:53:26.265098 5497 leader_election.cc:290] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:26.265239 5502 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:26.265458 5502 raft_consensus.cc:695] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [term 1 LEADER]: Becoming Leader. State: Replica: a861480e5a5d44f296ad166b772d4d38, State: Running, Role: LEADER
I20250905 07:53:26.265770 5502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } }
I20250905 07:53:26.266309 5497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:26.267638 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a861480e5a5d44f296ad166b772d4d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } } }
I20250905 07:53:26.267640 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a861480e5a5d44f296ad166b772d4d38. Latest consensus state: current_term: 1 leader_uuid: "a861480e5a5d44f296ad166b772d4d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a861480e5a5d44f296ad166b772d4d38" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43965 } } }
I20250905 07:53:26.267784 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:26.267783 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38 [sys.catalog]: This master's current role is: LEADER
W20250905 07:53:26.271978 5517 catalog_manager.cc:1560] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:26.272109 5517 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:26.272202 5512 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:26.273030 5512 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:26.278905 5512 catalog_manager.cc:1349] Generated new cluster ID: e4c796af238549cba4c75700767f3bbf
I20250905 07:53:26.278992 5512 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:26.284873 5512 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:26.285846 5512 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:26.293409 5512 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Generated new TSK 0
I20250905 07:53:26.294234 5512 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:26.344960 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43965
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:26.456523 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:26.456765 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:26.456835 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:26.460343 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:26.460427 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:26.460492 5525 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:26.460561 5525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:26.465070 5525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:26.466239 5525 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:26.467167 5525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:26.474051 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.474051 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.474051 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:26.474166 5525 server_base.cc:1047] running on GCE node
I20250905 07:53:26.474673 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:26.475226 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:26.476370 5525 hybrid_clock.cc:648] HybridClock initialized: now 1757058806476356 us; error 29 us; skew 500 ppm
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:26.479066 5525 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:26.480185 5525 webserver.cc:480] Webserver started at http://127.4.49.1:45053/ using document root <none> and password file <none>
I20250905 07:53:26.480759 5525 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:26.480810 5525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:26.480991 5525 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:26.482764 5525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "1c316d87a1c943028ec4e29a8fe44e11"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "8a18699b7ac6d43493ceca6c85534e6a"
server_key_iv: "9e29886f935e82a95532d442c4285cc7"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.483224 5525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "1c316d87a1c943028ec4e29a8fe44e11"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "8a18699b7ac6d43493ceca6c85534e6a"
server_key_iv: "9e29886f935e82a95532d442c4285cc7"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.486805 5525 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20250905 07:53:26.489051 5540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.490085 5525 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 07:53:26.490193 5525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "1c316d87a1c943028ec4e29a8fe44e11"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "8a18699b7ac6d43493ceca6c85534e6a"
server_key_iv: "9e29886f935e82a95532d442c4285cc7"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.490283 5525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:26.508908 5525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:26.511880 5525 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:26.512101 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:26.513577 5525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:26.513644 5525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.513700 5525 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:26.513728 5525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:26.524446 5525 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:41961
I20250905 07:53:26.524561 5657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:41961 every 8 connection(s)
I20250905 07:53:26.525664 5525 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:26.527154 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.514353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:45391 (local address 127.4.49.62:43965)
0905 07:53:26.514547 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.514552 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.515230 (+ 678us) server_negotiation.cc:408] Connection header received
0905 07:53:26.516320 (+ 1090us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.516324 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.516402 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.516464 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.518205 (+ 1741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.519036 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.519919 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.520718 (+ 799us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.522910 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.522935 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.522937 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.523062 (+ 125us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.524903 (+ 1841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.525532 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.525536 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.525538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.525606 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.525918 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.525922 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.525924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.526158 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.526255 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.526890 (+ 635us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.527025 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20250905 07:53:26.527844 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.514613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.1:45391)
0905 07:53:26.515071 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.515116 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.516038 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.516877 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.516887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.517292 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.518020 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.518041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.519193 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.519198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.519796 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.519806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.520039 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.520917 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.520956 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:26.522718 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:26.525046 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.525052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.525062 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.525397 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.525707 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.525710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.525711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.525828 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.526314 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:26.526321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:26.526608 (+ 287us) client_negotiation.cc:769] Sending connection context
0905 07:53:26.526833 (+ 225us) client_negotiation.cc:240] Negotiation successful
0905 07:53:26.527089 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20250905 07:53:26.529517 5658 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43965
I20250905 07:53:26.529726 5658 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:26.530347 5658 heartbeater.cc:507] Master 127.4.49.62:43965 requested a full tablet report, sending...
I20250905 07:53:26.531766 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5525
I20250905 07:53:26.531875 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:26.532161 4292 external_mini_cluster.cc:1468] Setting key a03243b150ecfe1eb9e4e046af796440
I20250905 07:53:26.532166 5449 ts_manager.cc:194] Registered new tserver with Master: 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961)
I20250905 07:53:26.533622 5449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:45391
I20250905 07:53:26.541726 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.531618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.1:41467)
0905 07:53:26.531917 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.531929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.532009 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.535615 (+ 3606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.535619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.535829 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.536097 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.536109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.537057 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.537061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.537483 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.537493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.537626 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.538166 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.538180 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:26.538489 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:26.540360 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.540363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.540365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.540547 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.540813 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.540816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.540818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.540867 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.541274 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:26.541276 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:26.541361 (+ 85us) client_negotiation.cc:769] Sending connection context
0905 07:53:26.541439 (+ 78us) client_negotiation.cc:240] Negotiation successful
0905 07:53:26.541557 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":124,"threads_started":1}
I20250905 07:53:26.541846 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.531695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41467 (local address 127.4.49.62:43965)
0905 07:53:26.535367 (+ 3672us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.535371 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.535386 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:53:26.535431 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.535433 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.535476 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.535556 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.536229 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.536926 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.537624 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.537836 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.538624 (+ 788us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.538652 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.538656 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.538701 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.540228 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.540644 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.540648 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.540651 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.540720 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.540956 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.540960 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.540962 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.541184 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.541290 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.541526 (+ 236us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.541663 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3565}
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:26.580013 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43965
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:26.690908 5670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:26.691170 5670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:26.691236 5670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:26.694712 5670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:26.694794 5670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:26.694850 5670 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:26.694950 5670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:26.699561 5670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:26.700749 5670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:26.701686 5670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:26.708554 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.708581 5678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.708554 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:26.708932 5670 server_base.cc:1047] running on GCE node
I20250905 07:53:26.709368 5670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:26.709961 5670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:26.711126 5670 hybrid_clock.cc:648] HybridClock initialized: now 1757058806711104 us; error 34 us; skew 500 ppm
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:26.714083 5670 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:26.715286 5670 webserver.cc:480] Webserver started at http://127.4.49.2:46711/ using document root <none> and password file <none>
I20250905 07:53:26.715862 5670 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:26.715914 5670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:26.716148 5670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:26.718050 5670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "b8f5863b426646cf84db47a4bb01e3fc"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "be4572ac94e0b22b81a4c7ffae88fb08"
server_key_iv: "266c92d1777e6d68595c955092ba45e3"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.718528 5670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "b8f5863b426646cf84db47a4bb01e3fc"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "be4572ac94e0b22b81a4c7ffae88fb08"
server_key_iv: "266c92d1777e6d68595c955092ba45e3"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.722399 5670 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 07:53:26.724931 5685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.726097 5670 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:26.726224 5670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "b8f5863b426646cf84db47a4bb01e3fc"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "be4572ac94e0b22b81a4c7ffae88fb08"
server_key_iv: "266c92d1777e6d68595c955092ba45e3"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.726325 5670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:26.753254 5670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:26.756280 5670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:26.756515 5670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:26.758085 5670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:26.758152 5670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.758204 5670 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:26.758232 5670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:26.768765 5670 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:42877
I20250905 07:53:26.768837 5802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:42877 every 8 connection(s)
I20250905 07:53:26.770056 5670 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:26.771689 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.759028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.2:43125)
0905 07:53:26.759495 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.759540 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.760352 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.760948 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.760960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.761535 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.762271 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.762290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.763381 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.763386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.764029 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.764039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.764285 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.765013 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.765056 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:26.766924 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:26.769064 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.769070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.769083 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.769401 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.769716 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.769719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.769721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.769841 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.770198 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:26.770205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:26.770523 (+ 318us) client_negotiation.cc:769] Sending connection context
0905 07:53:26.770772 (+ 249us) client_negotiation.cc:240] Negotiation successful
0905 07:53:26.771058 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20250905 07:53:26.772442 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.758736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:43125 (local address 127.4.49.62:43965)
0905 07:53:26.758881 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.758886 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.759654 (+ 768us) server_negotiation.cc:408] Connection header received
0905 07:53:26.760582 (+ 928us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.760587 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.760653 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.760750 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.762451 (+ 1701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.763238 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.764173 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.764386 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.767110 (+ 2724us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.767139 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.767142 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.767178 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.768941 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.769547 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.769551 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.769552 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.769612 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.769925 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.769927 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.769929 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.770092 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.770187 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.772194 (+ 2007us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.772318 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:53:26.773286 5803 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43965
I20250905 07:53:26.773455 5803 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:26.774109 5803 heartbeater.cc:507] Master 127.4.49.62:43965 requested a full tablet report, sending...
I20250905 07:53:26.775249 5449 ts_manager.cc:194] Registered new tserver with Master: b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2:42877)
I20250905 07:53:26.775969 5449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:43125
I20250905 07:53:26.776384 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5670
I20250905 07:53:26.776485 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:26.776705 4292 external_mini_cluster.cc:1468] Setting key 946f5886beca9801ab8eedd584a2d122
I20250905 07:53:26.782158 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.775691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:45553 (local address 127.4.49.62:43965)
0905 07:53:26.775808 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.775811 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.775914 (+ 103us) server_negotiation.cc:408] Connection header received
0905 07:53:26.776073 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.776077 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.776128 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.776199 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.776865 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.777506 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.778293 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.778468 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.779238 (+ 770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.779256 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.779260 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.779284 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.780734 (+ 1450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.781171 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.781175 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.781176 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.781220 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.781438 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.781440 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.781441 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.781602 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.781685 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.781933 (+ 248us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.782033 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:53:26.782233 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.775565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.2:45553)
0905 07:53:26.775836 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.775850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.775959 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.776233 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.776236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.776441 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.776711 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.776720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.777625 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.777628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.778165 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.778175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.778291 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.778830 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.778844 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:26.779124 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:26.780849 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.780854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.780856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.781077 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.781299 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.781302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.781305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.781367 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.781682 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:26.781685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:26.781787 (+ 102us) client_negotiation.cc:769] Sending connection context
0905 07:53:26.781899 (+ 112us) client_negotiation.cc:240] Negotiation successful
0905 07:53:26.782046 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:26.820977 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43965
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:26.929140 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:26.929386 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:26.929450 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:26.932934 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:26.933020 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:26.933077 5814 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:26.933123 5814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:26.937646 5814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:26.938768 5814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:26.939640 5814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:26.946476 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:26.946763 5814 server_base.cc:1047] running on GCE node
W20250905 07:53:26.946477 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:26.946642 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:26.947234 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:26.947837 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:26.948985 5814 hybrid_clock.cc:648] HybridClock initialized: now 1757058806948965 us; error 39 us; skew 500 ppm
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:26.951848 5814 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:26.953043 5814 webserver.cc:480] Webserver started at http://127.4.49.3:46723/ using document root <none> and password file <none>
I20250905 07:53:26.953632 5814 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:26.953691 5814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:26.953877 5814 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:26.955722 5814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "af1d872f033e48d5a792e60cbb72b152"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "88c515aa561452d59e6401e1996050f7"
server_key_iv: "cc45d184f4d67ad5f6ab3a2331016292"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.956256 5814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "af1d872f033e48d5a792e60cbb72b152"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "88c515aa561452d59e6401e1996050f7"
server_key_iv: "cc45d184f4d67ad5f6ab3a2331016292"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.959888 5814 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:53:26.962363 5831 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.963546 5814 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:26.963681 5814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "af1d872f033e48d5a792e60cbb72b152"
format_stamp: "Formatted at 2025-09-05 07:53:26 on dist-test-slave-mnfb"
server_key: "88c515aa561452d59e6401e1996050f7"
server_key_iv: "cc45d184f4d67ad5f6ab3a2331016292"
server_key_version: "encryptionkey@0"
I20250905 07:53:26.963790 5814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:26.974720 5814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:26.977576 5814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:26.977770 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:26.979212 5814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:26.979269 5814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:26.979324 5814 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:26.979349 5814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:26 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:26.990272 5814 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:46637
I20250905 07:53:26.990470 5949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:46637 every 8 connection(s)
I20250905 07:53:26.991461 5814 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:26.992799 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.979952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:49223 (local address 127.4.49.62:43965)
0905 07:53:26.980105 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.980109 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.980999 (+ 890us) server_negotiation.cc:408] Connection header received
0905 07:53:26.982146 (+ 1147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.982151 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.982218 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.982271 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.984421 (+ 2150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.985212 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.986036 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.986222 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.989036 (+ 2814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:26.989059 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:26.989061 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:26.989091 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:26.990670 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.991240 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.991242 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.991244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.991286 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:26.991579 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:26.991582 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:26.991584 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:26.991773 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:26.991871 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:26.992550 (+ 679us) server_negotiation.cc:300] Negotiation successful
0905 07:53:26.992674 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250905 07:53:26.993396 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.980361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.3:49223)
0905 07:53:26.980851 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.980889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.981730 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.982724 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.982736 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.983304 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.984198 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.984222 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.985380 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.985384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.985908 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.985919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.986149 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.986927 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.986970 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:26.988859 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:26.990796 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.990802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.990813 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.991137 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.991360 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:26.991363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:26.991365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:26.991486 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:26.991887 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:26.991893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:26.992221 (+ 328us) client_negotiation.cc:769] Sending connection context
0905 07:53:26.992468 (+ 247us) client_negotiation.cc:240] Negotiation successful
0905 07:53:26.992727 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20250905 07:53:26.994510 5950 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43965
I20250905 07:53:26.994724 5950 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:26.995270 5950 heartbeater.cc:507] Master 127.4.49.62:43965 requested a full tablet report, sending...
I20250905 07:53:26.996377 5450 ts_manager.cc:194] Registered new tserver with Master: af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637)
I20250905 07:53:26.996870 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 5814
I20250905 07:53:26.996961 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:26.997159 5450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:49223
I20250905 07:53:26.997201 4292 external_mini_cluster.cc:1468] Setting key a2ef3f807c3e78ffb44e2bcbb34a7add
I20250905 07:53:26.999259 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:53:27.002998 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.996574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.4.49.3:46693)
0905 07:53:26.996891 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:26.996902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:26.997007 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:26.997228 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:26.997231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:26.997439 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:26.997896 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.997904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.998733 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.998736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:26.999111 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:26.999119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.999216 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:26.999769 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:26.999785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:27.000119 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:27.001781 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:27.001784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:27.001786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:27.001954 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:27.002164 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:27.002166 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:27.002168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:27.002207 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:27.002537 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:27.002539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:27.002618 (+ 79us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.002694 (+ 76us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.002845 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20250905 07:53:27.003053 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:26.996645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:46693 (local address 127.4.49.62:43965)
0905 07:53:26.996767 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:26.996772 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:26.996970 (+ 198us) server_negotiation.cc:408] Connection header received
0905 07:53:26.997089 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:26.997092 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:26.997134 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:26.997197 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:26.998005 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.998624 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:26.999210 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:26.999397 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.000219 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:27.000237 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:27.000239 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:27.000263 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:27.001668 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.002038 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.002042 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.002044 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.002092 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.002268 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.002271 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.002273 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.002457 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:27.002542 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.002770 (+ 228us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.002892 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[5401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:27.025596 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.018012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.4.49.62:43965)
0905 07:53:27.018144 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.018148 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.018161 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:53:27.018204 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.018206 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.018252 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.018327 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:27.019114 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.019632 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.020316 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.020471 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.022606 (+ 2135us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:27.022630 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:27.022632 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:27.022665 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:27.024073 (+ 1408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.024497 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.024501 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.024502 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.024555 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.024789 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.024793 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.024795 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.024970 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:27.025097 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.025307 (+ 210us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.025434 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:53:27.028584 5450 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:27.031230 5450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:27.044569 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.039894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:46637 (local address 127.0.0.1:32942)
0905 07:53:27.040577 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.040619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.040764 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.041856 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.041859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.041879 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.042116 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.042123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.043280 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.043284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.044020 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.044029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.044164 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.044231 (+ 67us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.044322 (+ 91us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.044422 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":111,"threads_started":1}
I20250905 07:53:27.044775 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.040108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41961 (local address 127.0.0.1:52048)
0905 07:53:27.040645 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.040659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.040763 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.042009 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.042013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.042029 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.042410 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.042418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.043505 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.043508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.044415 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.044426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.044527 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.044541 (+ 14us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.044586 (+ 45us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.044645 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":90,"threads_started":1}
I20250905 07:53:27.045092 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.039934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.4.49.2:42877)
0905 07:53:27.040347 (+ 413us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.040368 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.040928 (+ 560us) server_negotiation.cc:408] Connection header received
0905 07:53:27.041009 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.041014 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.041204 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.041348 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.041812 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.042798 (+ 986us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.043927 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.044501 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.044625 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.044853 (+ 228us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.044957 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":83,"threads_started":1}
I20250905 07:53:27.045424 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.039760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:42877 (local address 127.0.0.1:36402)
0905 07:53:27.040792 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.040804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.040875 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.041348 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.041353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.041376 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.041700 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.041710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.042913 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.042917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.043773 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.043782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.044696 (+ 914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.044712 (+ 16us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.045241 (+ 529us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.045308 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":87,"threads_started":1}
I20250905 07:53:27.046001 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.040543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.4.49.1:41961)
0905 07:53:27.041662 (+ 1119us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.041668 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.041690 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:53:27.041768 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.041774 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.041904 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.042009 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.042542 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.043376 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.045122 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.045677 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.045771 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.045826 (+ 55us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.045898 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":92,"threads_started":1}
I20250905 07:53:27.048161 5735 tablet_service.cc:1468] Processing CreateTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f (DEFAULT_TABLE table=test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.048358 5590 tablet_service.cc:1468] Processing CreateTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f (DEFAULT_TABLE table=test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.049077 5735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707c093bdaa7453c89ed88ca7c8bcc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.049126 5590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707c093bdaa7453c89ed88ca7c8bcc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.052457 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.040547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.4.49.3:46637)
0905 07:53:27.041203 (+ 656us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.041209 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.041245 (+ 36us) server_negotiation.cc:408] Connection header received
0905 07:53:27.041327 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.041332 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.041529 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.041671 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.042241 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.043167 (+ 926us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.045134 (+ 1967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.051926 (+ 6792us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.052101 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.052182 (+ 81us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.052297 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":123,"threads_started":1}
I20250905 07:53:27.054710 5975 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Bootstrap starting.
I20250905 07:53:27.055943 5976 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Bootstrap starting.
I20250905 07:53:27.055573 5884 tablet_service.cc:1468] Processing CreateTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f (DEFAULT_TABLE table=test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.056649 5884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707c093bdaa7453c89ed88ca7c8bcc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.057241 5975 tablet_bootstrap.cc:654] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.058141 5976 tablet_bootstrap.cc:654] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.058362 5975 log.cc:826] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:27.058991 5976 log.cc:826] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:27.060230 5975 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: No bootstrap required, opened a new log
I20250905 07:53:27.060432 5975 ts_tablet_manager.cc:1397] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250905 07:53:27.060853 5976 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: No bootstrap required, opened a new log
I20250905 07:53:27.061045 5976 ts_tablet_manager.cc:1397] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:53:27.062994 5975 raft_consensus.cc:357] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.063246 5975 raft_consensus.cc:383] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.063311 5975 raft_consensus.cc:738] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5863b426646cf84db47a4bb01e3fc, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.063791 5975 consensus_queue.cc:260] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.063891 5979 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Bootstrap starting.
I20250905 07:53:27.063956 5976 raft_consensus.cc:357] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.064188 5976 raft_consensus.cc:383] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.064302 5976 raft_consensus.cc:738] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c316d87a1c943028ec4e29a8fe44e11, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.064596 5803 heartbeater.cc:499] Master 127.4.49.62:43965 was elected leader, sending a full tablet report...
I20250905 07:53:27.064850 5975 ts_tablet_manager.cc:1428] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 07:53:27.064823 5976 consensus_queue.cc:260] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.065447 5658 heartbeater.cc:499] Master 127.4.49.62:43965 was elected leader, sending a full tablet report...
I20250905 07:53:27.065635 5976 ts_tablet_manager.cc:1428] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250905 07:53:27.066255 5979 tablet_bootstrap.cc:654] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.067523 5979 log.cc:826] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:27.070058 5979 tablet_bootstrap.cc:492] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: No bootstrap required, opened a new log
I20250905 07:53:27.070262 5979 ts_tablet_manager.cc:1397] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250905 07:53:27.072769 5979 raft_consensus.cc:357] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.072997 5979 raft_consensus.cc:383] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.073069 5979 raft_consensus.cc:738] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1d872f033e48d5a792e60cbb72b152, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.073523 5979 consensus_queue.cc:260] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.074234 5950 heartbeater.cc:499] Master 127.4.49.62:43965 was elected leader, sending a full tablet report...
I20250905 07:53:27.074374 5979 ts_tablet_manager.cc:1428] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20250905 07:53:27.086046 5980 raft_consensus.cc:491] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:27.086277 5980 raft_consensus.cc:513] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.087371 5980 leader_election.cc:290] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637), 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961)
I20250905 07:53:27.090447 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.087640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:46637 (local address 127.4.49.2:35277)
0905 07:53:27.087800 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.087816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.087914 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.088246 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.088249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.088270 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.088531 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.088539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.089445 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.089449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.090107 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.090115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.090209 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.090228 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.090262 (+ 34us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.090308 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"spinlock_wait_cycles":14720}
I20250905 07:53:27.090706 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.087653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41961 (local address 127.4.49.2:41735)
0905 07:53:27.088172 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.088183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.088321 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.088599 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.088603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.088621 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.088901 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.088911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.089667 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.089670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.090299 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.090308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.090415 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.090434 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.090481 (+ 47us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.090535 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":134,"thread_start_us":152,"threads_started":1}
I20250905 07:53:27.090998 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.087734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:35277 (local address 127.4.49.3:46637)
0905 07:53:27.087860 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.087864 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.087882 (+ 18us) server_negotiation.cc:408] Connection header received
0905 07:53:27.088032 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.088036 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.088119 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.088222 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.088677 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.089333 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.090232 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.090728 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.090769 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.090820 (+ 51us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.090883 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250905 07:53:27.091168 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.087748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:41735 (local address 127.4.49.1:41961)
0905 07:53:27.087864 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.087868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.088247 (+ 379us) server_negotiation.cc:408] Connection header received
0905 07:53:27.088444 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.088448 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.088503 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.088571 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.089007 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.089548 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.090432 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.090907 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.090945 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.090996 (+ 51us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.091052 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:53:27.091581 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "707c093bdaa7453c89ed88ca7c8bcc4f" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152" is_pre_election: true
I20250905 07:53:27.091692 5610 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "707c093bdaa7453c89ed88ca7c8bcc4f" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c316d87a1c943028ec4e29a8fe44e11" is_pre_election: true
I20250905 07:53:27.091885 5904 raft_consensus.cc:2466] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 0.
I20250905 07:53:27.091928 5610 raft_consensus.cc:2466] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 0.
I20250905 07:53:27.092404 5688 leader_election.cc:304] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c316d87a1c943028ec4e29a8fe44e11, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.092645 5980 raft_consensus.cc:2802] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:27.092720 5980 raft_consensus.cc:491] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:27.092767 5980 raft_consensus.cc:3058] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.093716 5980 raft_consensus.cc:513] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.094066 5980 leader_election.cc:290] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 election: Requested vote from peers af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637), 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961)
I20250905 07:53:27.094548 5610 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "707c093bdaa7453c89ed88ca7c8bcc4f" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c316d87a1c943028ec4e29a8fe44e11"
I20250905 07:53:27.094673 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "707c093bdaa7453c89ed88ca7c8bcc4f" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152"
I20250905 07:53:27.094704 5610 raft_consensus.cc:3058] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.094800 5904 raft_consensus.cc:3058] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.096252 5904 raft_consensus.cc:2466] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 1.
I20250905 07:53:27.096640 5610 raft_consensus.cc:2466] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 1.
I20250905 07:53:27.096679 5689 leader_election.cc:304] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1d872f033e48d5a792e60cbb72b152, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.096882 5980 raft_consensus.cc:2802] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:27.097106 5980 raft_consensus.cc:695] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 1 LEADER]: Becoming Leader. State: Replica: b8f5863b426646cf84db47a4bb01e3fc, State: Running, Role: LEADER
I20250905 07:53:27.097497 5980 consensus_queue.cc:237] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } }
I20250905 07:53:27.101040 5449 catalog_manager.cc:5582] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2). New cstate: current_term: 1 leader_uuid: "b8f5863b426646cf84db47a4bb01e3fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:27.128731 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.125570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.4.49.2:42877)
0905 07:53:27.125720 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.125725 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.125742 (+ 17us) server_negotiation.cc:408] Connection header received
0905 07:53:27.125908 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.125911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.125968 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.126050 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:27.126394 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.126912 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.127555 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.127760 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.127818 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.128316 (+ 498us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.128421 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.128526 (+ 105us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.128584 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:53:27.134274 5610 raft_consensus.cc:1273] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Refusing update from remote peer b8f5863b426646cf84db47a4bb01e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.134274 5904 raft_consensus.cc:1273] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Refusing update from remote peer b8f5863b426646cf84db47a4bb01e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.134894 5985 consensus_queue.cc:1035] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.135089 5980 consensus_queue.cc:1035] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.142138 5988 mvcc.cc:204] Tried to move back new op lower bound from 7196912874015309824 to 7196912873874018304. Current Snapshot: MvccSnapshot[applied={T|T < 7196912874015309824}]
I20250905 07:53:27.142437 5990 mvcc.cc:204] Tried to move back new op lower bound from 7196912874015309824 to 7196912873874018304. Current Snapshot: MvccSnapshot[applied={T|T < 7196912874015309824}]
I20250905 07:53:27.142670 5989 mvcc.cc:204] Tried to move back new op lower bound from 7196912874015309824 to 7196912873874018304. Current Snapshot: MvccSnapshot[applied={T|T < 7196912874015309824}]
I20250905 07:53:27.153589 5450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:27.153745 5450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:27.156230 5450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Sending DeleteTablet for 3 replicas of tablet 707c093bdaa7453c89ed88ca7c8bcc4f
I20250905 07:53:27.157058 5735 tablet_service.cc:1515] Processing DeleteTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:36402
I20250905 07:53:27.157088 5884 tablet_service.cc:1515] Processing DeleteTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:32942
I20250905 07:53:27.157157 5590 tablet_service.cc:1515] Processing DeleteTablet for tablet 707c093bdaa7453c89ed88ca7c8bcc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:52048
I20250905 07:53:27.157692 5997 tablet_replica.cc:331] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: stopping tablet replica
I20250905 07:53:27.157838 5999 tablet_replica.cc:331] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: stopping tablet replica
I20250905 07:53:27.158100 5999 raft_consensus.cc:2241] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:27.158103 5997 raft_consensus.cc:2241] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:27.158481 5999 raft_consensus.cc:2270] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.158689 5997 raft_consensus.cc:2270] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.158614 5450 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:27.159094 5450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:27.159036 5998 tablet_replica.cc:331] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: stopping tablet replica
I20250905 07:53:27.159415 5998 raft_consensus.cc:2241] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:27.159737 5998 raft_consensus.cc:2270] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.160431 5997 ts_tablet_manager.cc:1905] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.161085 5999 ts_tablet_manager.cc:1905] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.161680 5998 ts_tablet_manager.cc:1905] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.164000 5999 ts_tablet_manager.cc:1918] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:27.164232 5999 log.cc:1199] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/707c093bdaa7453c89ed88ca7c8bcc4f
I20250905 07:53:27.164640 5999 ts_tablet_manager.cc:1939] T 707c093bdaa7453c89ed88ca7c8bcc4f P 1c316d87a1c943028ec4e29a8fe44e11: Deleting consensus metadata
I20250905 07:53:27.165366 5998 ts_tablet_manager.cc:1918] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:27.165542 5998 log.cc:1199] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/707c093bdaa7453c89ed88ca7c8bcc4f
I20250905 07:53:27.165586 5590 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d (DEFAULT_TABLE table=test-table [id=639c722032ed4b93a09100ace9474827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.165904 5590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec480a7fea74bbfa5dd813e438e865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.166049 5998 ts_tablet_manager.cc:1939] T 707c093bdaa7453c89ed88ca7c8bcc4f P af1d872f033e48d5a792e60cbb72b152: Deleting consensus metadata
I20250905 07:53:27.166154 5435 catalog_manager.cc:4928] TS 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961): tablet 707c093bdaa7453c89ed88ca7c8bcc4f (table test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]) successfully deleted
I20250905 07:53:27.166812 5884 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d (DEFAULT_TABLE table=test-table [id=639c722032ed4b93a09100ace9474827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.167079 5884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec480a7fea74bbfa5dd813e438e865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.168386 5976 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Bootstrap starting.
I20250905 07:53:27.168243 5735 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d (DEFAULT_TABLE table=test-table [id=639c722032ed4b93a09100ace9474827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:27.168951 5735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec480a7fea74bbfa5dd813e438e865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.169492 5976 tablet_bootstrap.cc:654] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.169646 5979 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Bootstrap starting.
I20250905 07:53:27.170333 5436 catalog_manager.cc:4928] TS af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637): tablet 707c093bdaa7453c89ed88ca7c8bcc4f (table test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]) successfully deleted
I20250905 07:53:27.170667 5979 tablet_bootstrap.cc:654] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.171355 5976 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: No bootstrap required, opened a new log
I20250905 07:53:27.171499 5976 ts_tablet_manager.cc:1397] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:27.172046 5976 raft_consensus.cc:357] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.172180 5975 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Bootstrap starting.
I20250905 07:53:27.172204 5976 raft_consensus.cc:383] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.172233 5976 raft_consensus.cc:738] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c316d87a1c943028ec4e29a8fe44e11, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.172345 5976 consensus_queue.cc:260] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.172595 5976 ts_tablet_manager.cc:1428] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:27.173235 5975 tablet_bootstrap.cc:654] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.174111 5997 ts_tablet_manager.cc:1918] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:27.174254 5997 log.cc:1199] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/707c093bdaa7453c89ed88ca7c8bcc4f
I20250905 07:53:27.174609 5997 ts_tablet_manager.cc:1939] T 707c093bdaa7453c89ed88ca7c8bcc4f P b8f5863b426646cf84db47a4bb01e3fc: Deleting consensus metadata
I20250905 07:53:27.175750 5975 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: No bootstrap required, opened a new log
I20250905 07:53:27.175837 5975 ts_tablet_manager.cc:1397] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:53:27.175968 5433 catalog_manager.cc:4928] TS b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2:42877): tablet 707c093bdaa7453c89ed88ca7c8bcc4f (table test-table [id=f9b8ae2c0a7e4ee2ab3a78f27c59840b]) successfully deleted
I20250905 07:53:27.176390 5979 tablet_bootstrap.cc:492] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: No bootstrap required, opened a new log
I20250905 07:53:27.176396 5975 raft_consensus.cc:357] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.176491 5979 ts_tablet_manager.cc:1397] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250905 07:53:27.176501 5975 raft_consensus.cc:383] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.176533 5975 raft_consensus.cc:738] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5863b426646cf84db47a4bb01e3fc, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.176664 5975 consensus_queue.cc:260] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.176913 5975 ts_tablet_manager.cc:1428] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:27.176977 5979 raft_consensus.cc:357] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.177071 5979 raft_consensus.cc:383] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.177100 5979 raft_consensus.cc:738] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1d872f033e48d5a792e60cbb72b152, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.177211 5979 consensus_queue.cc:260] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.177490 5979 ts_tablet_manager.cc:1428] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:27.235106 5985 raft_consensus.cc:491] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:27.235265 5985 raft_consensus.cc:513] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.235744 5985 leader_election.cc:290] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961), af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637)
I20250905 07:53:27.236253 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ec480a7fea74bbfa5dd813e438e865d" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152" is_pre_election: true
I20250905 07:53:27.236294 5609 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ec480a7fea74bbfa5dd813e438e865d" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c316d87a1c943028ec4e29a8fe44e11" is_pre_election: true
I20250905 07:53:27.236414 5904 raft_consensus.cc:2466] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 0.
I20250905 07:53:27.236459 5609 raft_consensus.cc:2466] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 0.
I20250905 07:53:27.236747 5689 leader_election.cc:304] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1d872f033e48d5a792e60cbb72b152, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.236960 5985 raft_consensus.cc:2802] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:27.237027 5985 raft_consensus.cc:491] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:27.237061 5985 raft_consensus.cc:3058] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.238345 5985 raft_consensus.cc:513] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.238759 5985 leader_election.cc:290] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 election: Requested vote from peers 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961), af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637)
I20250905 07:53:27.239207 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ec480a7fea74bbfa5dd813e438e865d" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152"
I20250905 07:53:27.239207 5609 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ec480a7fea74bbfa5dd813e438e865d" candidate_uuid: "b8f5863b426646cf84db47a4bb01e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c316d87a1c943028ec4e29a8fe44e11"
I20250905 07:53:27.239348 5609 raft_consensus.cc:3058] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.239349 5904 raft_consensus.cc:3058] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.240525 5904 raft_consensus.cc:2466] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 1.
I20250905 07:53:27.240525 5609 raft_consensus.cc:2466] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5863b426646cf84db47a4bb01e3fc in term 1.
I20250905 07:53:27.240939 5689 leader_election.cc:304] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1d872f033e48d5a792e60cbb72b152, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.241163 5985 raft_consensus.cc:2802] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:27.241289 5985 raft_consensus.cc:695] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 1 LEADER]: Becoming Leader. State: Replica: b8f5863b426646cf84db47a4bb01e3fc, State: Running, Role: LEADER
I20250905 07:53:27.241443 5985 consensus_queue.cc:237] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
W20250905 07:53:27.242846 5951 tablet.cc:2378] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:27.243582 5450 catalog_manager.cc:5582] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2). New cstate: current_term: 1 leader_uuid: "b8f5863b426646cf84db47a4bb01e3fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } health_report { overall_health: UNKNOWN } } }
W20250905 07:53:27.271571 5804 tablet.cc:2378] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:53:27.277118 5659 tablet.cc:2378] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058806, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:27.295578 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.285963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43965 (local address 127.0.0.1:56712)
0905 07:53:27.286277 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.286294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.286428 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.286751 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.286754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.287332 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:27.287608 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.287619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.288447 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.288452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.289030 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.289042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.289181 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.289826 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:27.289844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:27.291649 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:27.293683 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:27.293696 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:27.293703 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:27.294106 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:27.294422 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:27.294425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:27.294428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:27.294495 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:27.294875 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:27.294883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:27.295025 (+ 142us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.295182 (+ 157us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.295369 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20250905 07:53:27.295794 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.286097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.4.49.62:43965)
0905 07:53:27.286223 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.286227 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.286390 (+ 163us) server_negotiation.cc:408] Connection header received
0905 07:53:27.286567 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.286571 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.286619 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.286694 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:27.287763 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.288307 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.289169 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.289306 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.291876 (+ 2570us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:27.291903 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:27.291906 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:27.291935 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:27.293532 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.294239 (+ 707us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.294242 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.294244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.294294 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.294581 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.294583 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.294585 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.294755 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:27.294830 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.295235 (+ 405us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.295523 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":49}
I20250905 07:53:27.300303 5450 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:56712:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250905 07:53:27.301242 5450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:27.307188 5590 tablet_service.cc:1468] Processing CreateTablet for tablet 672b3a62d3e341bf8c3221ec602256eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74d0ed6b079840dcbd43c989d91df56d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:27.307384 5735 tablet_service.cc:1468] Processing CreateTablet for tablet 672b3a62d3e341bf8c3221ec602256eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74d0ed6b079840dcbd43c989d91df56d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:27.307425 5884 tablet_service.cc:1468] Processing CreateTablet for tablet 672b3a62d3e341bf8c3221ec602256eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74d0ed6b079840dcbd43c989d91df56d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:27.307665 5590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b3a62d3e341bf8c3221ec602256eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.307685 5735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b3a62d3e341bf8c3221ec602256eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.307832 5884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b3a62d3e341bf8c3221ec602256eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:27.310516 5975 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: Bootstrap starting.
I20250905 07:53:27.310771 5979 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: Bootstrap starting.
I20250905 07:53:27.311662 5975 tablet_bootstrap.cc:654] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.311730 5979 tablet_bootstrap.cc:654] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.312450 5976 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: Bootstrap starting.
I20250905 07:53:27.313766 5975 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: No bootstrap required, opened a new log
I20250905 07:53:27.313877 5975 ts_tablet_manager.cc:1397] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:27.313897 5976 tablet_bootstrap.cc:654] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:27.314412 5975 raft_consensus.cc:357] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.314538 5975 raft_consensus.cc:383] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.314568 5975 raft_consensus.cc:738] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5863b426646cf84db47a4bb01e3fc, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.314711 5975 consensus_queue.cc:260] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.315035 5985 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.315176 5985 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.315044 5975 ts_tablet_manager.cc:1428] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 07:53:27.316319 5979 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: No bootstrap required, opened a new log
I20250905 07:53:27.316430 5979 ts_tablet_manager.cc:1397] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250905 07:53:27.316615 5976 tablet_bootstrap.cc:492] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: No bootstrap required, opened a new log
I20250905 07:53:27.316708 5976 ts_tablet_manager.cc:1397] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250905 07:53:27.317005 5979 raft_consensus.cc:357] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.317137 5979 raft_consensus.cc:383] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.317178 5979 raft_consensus.cc:738] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1d872f033e48d5a792e60cbb72b152, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.317363 5979 consensus_queue.cc:260] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.317411 5976 raft_consensus.cc:357] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.317524 5976 raft_consensus.cc:383] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:27.317562 5976 raft_consensus.cc:738] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c316d87a1c943028ec4e29a8fe44e11, State: Initialized, Role: FOLLOWER
I20250905 07:53:27.317667 5979 ts_tablet_manager.cc:1428] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:27.317914 5983 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.318034 5983 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.318243 5976 consensus_queue.cc:260] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.318532 5976 ts_tablet_manager.cc:1428] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:27.318572 5981 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.318708 5981 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.341022 5981 raft_consensus.cc:491] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:27.341179 5981 raft_consensus.cc:513] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.342304 5981 leader_election.cc:290] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2:42877), af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637)
I20250905 07:53:27.345597 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.342550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:42877 (local address 127.4.49.1:55023)
0905 07:53:27.343028 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.343042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.343149 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.343409 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.343412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.343428 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.343699 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.343706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.344452 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.344456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.345081 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.345090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.345205 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.345231 (+ 26us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.345282 (+ 51us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.345360 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":176,"threads_started":1}
I20250905 07:53:27.345602 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.342591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:46637 (local address 127.4.49.1:47523)
0905 07:53:27.343106 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.343121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.343238 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.343482 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.343485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.343499 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:27.343739 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.343744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.344470 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.344473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.345093 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.345100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.345226 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.345247 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.345297 (+ 50us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.345356 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":140,"threads_started":1}
I20250905 07:53:27.345963 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.342707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:55023 (local address 127.4.49.2:42877)
0905 07:53:27.342981 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.342984 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.343127 (+ 143us) server_negotiation.cc:408] Connection header received
0905 07:53:27.343248 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.343251 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.343303 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.343369 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.343821 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.344341 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.345205 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.345682 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.345726 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.345783 (+ 57us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.345849 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20250905 07:53:27.345981 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.342840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:47523 (local address 127.4.49.3:46637)
0905 07:53:27.342985 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.342989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.343217 (+ 228us) server_negotiation.cc:408] Connection header received
0905 07:53:27.343332 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.343334 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.343384 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.343462 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:27.343851 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.344363 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.345210 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.345704 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.345747 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.345804 (+ 57us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.345864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:53:27.346498 5756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "672b3a62d3e341bf8c3221ec602256eb" candidate_uuid: "1c316d87a1c943028ec4e29a8fe44e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5863b426646cf84db47a4bb01e3fc" is_pre_election: true
I20250905 07:53:27.346524 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "672b3a62d3e341bf8c3221ec602256eb" candidate_uuid: "1c316d87a1c943028ec4e29a8fe44e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152" is_pre_election: true
I20250905 07:53:27.346678 5904 raft_consensus.cc:2466] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c316d87a1c943028ec4e29a8fe44e11 in term 0.
I20250905 07:53:27.346715 5756 raft_consensus.cc:2466] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c316d87a1c943028ec4e29a8fe44e11 in term 0.
I20250905 07:53:27.347124 5541 leader_election.cc:304] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c316d87a1c943028ec4e29a8fe44e11, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.347393 5981 raft_consensus.cc:2802] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:27.347456 5981 raft_consensus.cc:491] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:27.347496 5981 raft_consensus.cc:3058] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.348500 5981 raft_consensus.cc:513] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.348882 5981 leader_election.cc:290] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [CANDIDATE]: Term 1 election: Requested vote from peers b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2:42877), af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637)
I20250905 07:53:27.349306 5904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "672b3a62d3e341bf8c3221ec602256eb" candidate_uuid: "1c316d87a1c943028ec4e29a8fe44e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1d872f033e48d5a792e60cbb72b152"
I20250905 07:53:27.349448 5904 raft_consensus.cc:3058] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.349454 5756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "672b3a62d3e341bf8c3221ec602256eb" candidate_uuid: "1c316d87a1c943028ec4e29a8fe44e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5863b426646cf84db47a4bb01e3fc"
I20250905 07:53:27.349565 5756 raft_consensus.cc:3058] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:27.350623 5756 raft_consensus.cc:2466] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c316d87a1c943028ec4e29a8fe44e11 in term 1.
I20250905 07:53:27.350623 5904 raft_consensus.cc:2466] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c316d87a1c943028ec4e29a8fe44e11 in term 1.
I20250905 07:53:27.351130 5541 leader_election.cc:304] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c316d87a1c943028ec4e29a8fe44e11, b8f5863b426646cf84db47a4bb01e3fc; no voters:
I20250905 07:53:27.351359 5981 raft_consensus.cc:2802] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:27.351608 5981 raft_consensus.cc:695] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 LEADER]: Becoming Leader. State: Replica: 1c316d87a1c943028ec4e29a8fe44e11, State: Running, Role: LEADER
I20250905 07:53:27.351886 5981 consensus_queue.cc:237] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } }
I20250905 07:53:27.353174 6015 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: TxnStatusTablet state changed. Reason: New leader 1c316d87a1c943028ec4e29a8fe44e11. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.353324 6015 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:27.353649 6017 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 07:53:27.353806 6017 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250905 07:53:27.354594 5447 catalog_manager.cc:5582] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:27.375924 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.372420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.4.49.1:41961)
0905 07:53:27.372535 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.372538 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.372553 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:53:27.372769 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.372771 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.372831 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.372906 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:27.373307 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.373829 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.374828 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.374974 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.375027 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.375526 (+ 499us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.375623 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.375744 (+ 121us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.375801 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250905 07:53:27.375972 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.372300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41961 (local address 127.0.0.1:52064)
0905 07:53:27.372475 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.372492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.372623 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.372931 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.372935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.372944 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:27.373206 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.373214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.373933 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.373937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.374696 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.374710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.374841 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.374903 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.375631 (+ 728us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.375657 (+ 26us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.375729 (+ 72us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.375822 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20250905 07:53:27.379149 5904 raft_consensus.cc:1273] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Refusing update from remote peer 1c316d87a1c943028ec4e29a8fe44e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.379323 5756 raft_consensus.cc:1273] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Refusing update from remote peer 1c316d87a1c943028ec4e29a8fe44e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.379745 5981 consensus_queue.cc:1035] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.379935 6015 consensus_queue.cc:1035] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.381803 5983 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: TxnStatusTablet state changed. Reason: New leader 1c316d87a1c943028ec4e29a8fe44e11. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.381942 5983 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.382526 5985 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: TxnStatusTablet state changed. Reason: New leader 1c316d87a1c943028ec4e29a8fe44e11. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.382627 5985 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.383769 6015 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.383882 6015 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:27.384615 5981 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.384714 5981 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P 1c316d87a1c943028ec4e29a8fe44e11: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:27.385761 5985 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.385871 5985 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P b8f5863b426646cf84db47a4bb01e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.386583 5983 tablet_replica.cc:440] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c316d87a1c943028ec4e29a8fe44e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 } } peers { permanent_uuid: "b8f5863b426646cf84db47a4bb01e3fc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 42877 } } peers { permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 } } }
I20250905 07:53:27.386710 5983 tablet_replica.cc:443] T 672b3a62d3e341bf8c3221ec602256eb P af1d872f033e48d5a792e60cbb72b152: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:27.399832 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.396928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41961 (local address 127.4.49.2:42815)
0905 07:53:27.397339 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.397351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.397469 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.397712 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.397714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.397724 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:27.397944 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.397950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.398634 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.398637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.399049 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.399056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.399159 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.399211 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.399550 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.399556 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.399611 (+ 55us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.399702 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20250905 07:53:27.399828 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.397051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:42815 (local address 127.4.49.1:41961)
0905 07:53:27.397194 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.397199 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.397426 (+ 227us) server_negotiation.cc:408] Connection header received
0905 07:53:27.397566 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.397569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.397626 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.397690 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:27.398051 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.398521 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.399156 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.399336 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.399388 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.399467 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.399528 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.399628 (+ 100us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.399681 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250905 07:53:27.406499 5609 raft_consensus.cc:1273] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Refusing update from remote peer b8f5863b426646cf84db47a4bb01e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.406502 5904 raft_consensus.cc:1273] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Refusing update from remote peer b8f5863b426646cf84db47a4bb01e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:27.406985 5980 consensus_queue.cc:1035] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c316d87a1c943028ec4e29a8fe44e11" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.407153 5985 consensus_queue.cc:1035] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1d872f033e48d5a792e60cbb72b152" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:27.429890 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.426229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:37973 (local address 127.4.49.2:42877)
0905 07:53:27.426360 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.426364 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.426765 (+ 401us) server_negotiation.cc:408] Connection header received
0905 07:53:27.426972 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.426976 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.427030 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.427110 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:27.427524 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.428156 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.429071 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.429275 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.429331 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.429447 (+ 116us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.429522 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.429687 (+ 165us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.429748 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:53:27.429921 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.426128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:42877 (local address 127.4.49.1:37973)
0905 07:53:27.426665 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:27.426678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:27.426815 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:27.427133 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:27.427136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:27.427145 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:27.427410 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.427417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.428266 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.428270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:27.428946 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:27.428957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.429076 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.429135 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:27.429540 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:27.429546 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:53:27.429649 (+ 103us) client_negotiation.cc:240] Negotiation successful
0905 07:53:27.429730 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":137,"threads_started":1}
I20250905 07:53:27.452128 5447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:27.452311 5447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:27.454416 5447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a861480e5a5d44f296ad166b772d4d38: Sending DeleteTablet for 3 replicas of tablet 6ec480a7fea74bbfa5dd813e438e865d
I20250905 07:53:27.455000 5735 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:36402
I20250905 07:53:27.455013 5590 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:52048
I20250905 07:53:27.455127 5884 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ec480a7fea74bbfa5dd813e438e865d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:27 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:32942
I20250905 07:53:27.455194 5997 tablet_replica.cc:331] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: stopping tablet replica
I20250905 07:53:27.455303 5997 raft_consensus.cc:2241] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:27.455327 5999 tablet_replica.cc:331] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: stopping tablet replica
I20250905 07:53:27.455332 5998 tablet_replica.cc:331] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: stopping tablet replica
I20250905 07:53:27.455425 5999 raft_consensus.cc:2241] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:27.455457 5998 raft_consensus.cc:2241] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:27.455523 5999 raft_consensus.cc:2270] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.455514 5997 raft_consensus.cc:2270] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.455574 5998 raft_consensus.cc:2270] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:27.456393 5999 ts_tablet_manager.cc:1905] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.456393 5997 ts_tablet_manager.cc:1905] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.456542 5998 ts_tablet_manager.cc:1905] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:27.459136 5999 ts_tablet_manager.cc:1918] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:27.459136 5997 ts_tablet_manager.cc:1918] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:27.459136 5998 ts_tablet_manager.cc:1918] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:27.459218 5999 log.cc:1199] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/6ec480a7fea74bbfa5dd813e438e865d
I20250905 07:53:27.459218 5997 log.cc:1199] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/6ec480a7fea74bbfa5dd813e438e865d
I20250905 07:53:27.459218 5998 log.cc:1199] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/6ec480a7fea74bbfa5dd813e438e865d
I20250905 07:53:27.459474 5997 ts_tablet_manager.cc:1939] T 6ec480a7fea74bbfa5dd813e438e865d P b8f5863b426646cf84db47a4bb01e3fc: Deleting consensus metadata
I20250905 07:53:27.459472 5999 ts_tablet_manager.cc:1939] T 6ec480a7fea74bbfa5dd813e438e865d P 1c316d87a1c943028ec4e29a8fe44e11: Deleting consensus metadata
I20250905 07:53:27.459484 5998 ts_tablet_manager.cc:1939] T 6ec480a7fea74bbfa5dd813e438e865d P af1d872f033e48d5a792e60cbb72b152: Deleting consensus metadata
I20250905 07:53:27.460304 5436 catalog_manager.cc:4928] TS af1d872f033e48d5a792e60cbb72b152 (127.4.49.3:46637): tablet 6ec480a7fea74bbfa5dd813e438e865d (table test-table [id=639c722032ed4b93a09100ace9474827]) successfully deleted
I20250905 07:53:27.460539 5435 catalog_manager.cc:4928] TS 1c316d87a1c943028ec4e29a8fe44e11 (127.4.49.1:41961): tablet 6ec480a7fea74bbfa5dd813e438e865d (table test-table [id=639c722032ed4b93a09100ace9474827]) successfully deleted
I20250905 07:53:27.461601 5433 catalog_manager.cc:4928] TS b8f5863b426646cf84db47a4bb01e3fc (127.4.49.2:42877): tablet 6ec480a7fea74bbfa5dd813e438e865d (table test-table [id=639c722032ed4b93a09100ace9474827]) successfully deleted
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[5401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:27.470655 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.461750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.4.49.1:41961)
0905 07:53:27.461873 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.461878 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.462259 (+ 381us) server_negotiation.cc:408] Connection header received
0905 07:53:27.462366 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.462369 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.462430 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.462519 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:27.463359 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.464141 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.464740 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.464929 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.467162 (+ 2233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:27.467192 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:27.467238 (+ 46us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:27.467272 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:27.469129 (+ 1857us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.469605 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.469609 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.469611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.469665 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.469859 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.469861 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.469863 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.470042 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:27.470211 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.470376 (+ 165us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.470502 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20250905 07:53:27.471149 5652 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52076
I20250905 07:53:27.479938 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:27.473112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.4.49.62:43965)
0905 07:53:27.473242 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:27.473246 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:27.473262 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:53:27.473365 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:27.473368 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:27.473432 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:27.473510 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:27.474279 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.475087 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:27.475725 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:27.475921 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:27.476966 (+ 1045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:27.476991 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:27.476995 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:27.477026 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:27.478539 (+ 1513us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.478937 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.478941 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.478943 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.478992 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:27.479152 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:27.479155 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:27.479156 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:27.479346 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:27.479494 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:27.479644 (+ 150us) server_negotiation.cc:300] Negotiation successful
0905 07:53:27.479778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20250905 07:53:27.480393 5447 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56724
I20250905 07:53:27.481850 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5525
I20250905 07:53:27.488581 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5670
I20250905 07:53:27.495061 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5814
I20250905 07:53:27.500622 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 5417
2025-09-05T07:53:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3408 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[6038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[6038](info): set up 2 sockets
Sep 05 07:53:27 dist-test-slave-mnfb krb5kdc[6038](info): commencing operation
krb5kdc: starting...
W20250905 07:53:29.342483 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:29 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:29.471889 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43081
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:29.589833 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:29.590107 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:29.590184 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:29.593568 6054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:29.593652 6054 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250905 07:53:29.593677 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:29.593706 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:29.593735 6054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:29.593770 6054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:29.598202 6054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:43081
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:29.599332 6054 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:29.600265 6054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:29.605768 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:29.605950 6054 server_base.cc:1047] running on GCE node
W20250905 07:53:29.605767 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:29.605773 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:29.606622 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:29.607543 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:29.608703 6054 hybrid_clock.cc:648] HybridClock initialized: now 1757058809608685 us; error 31 us; skew 500 ppm
Sep 05 07:53:29 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:29.611986 6054 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:29.613097 6054 webserver.cc:480] Webserver started at http://127.4.49.62:42961/ using document root <none> and password file <none>
I20250905 07:53:29.613641 6054 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:29.613690 6054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:29.613874 6054 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:29.615586 6054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "1b6a77ef2a19499189ac3e3aede2789c"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "601c0839ee3936dc0ac71489721674bc"
server_key_iv: "c4f8d961a6192faeb150bda8cd168bb9"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.616044 6054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "1b6a77ef2a19499189ac3e3aede2789c"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "601c0839ee3936dc0ac71489721674bc"
server_key_iv: "c4f8d961a6192faeb150bda8cd168bb9"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.619470 6054 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:29.621852 6069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:29.622898 6054 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250905 07:53:29.623003 6054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "1b6a77ef2a19499189ac3e3aede2789c"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "601c0839ee3936dc0ac71489721674bc"
server_key_iv: "c4f8d961a6192faeb150bda8cd168bb9"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.623106 6054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:29.645907 6054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:29.649108 6054 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:29.649282 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:29.657506 6054 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:43081
I20250905 07:53:29.657503 6133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:43081 every 8 connection(s)
I20250905 07:53:29.658587 6054 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:29.661749 6134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:29.666828 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Bootstrap starting.
I20250905 07:53:29.667745 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6054
I20250905 07:53:29.667872 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:29.668118 4292 external_mini_cluster.cc:1468] Setting key 4a362213c4131cf620ed3ea3583c5e96
I20250905 07:53:29.669533 6134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:29.670239 6134 log.cc:826] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:29.672379 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: No bootstrap required, opened a new log
I20250905 07:53:29.675556 6134 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } }
Sep 05 07:53:29 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:29.675778 6134 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:29.675856 6134 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6a77ef2a19499189ac3e3aede2789c, State: Initialized, Role: FOLLOWER
I20250905 07:53:29.676337 6134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } }
I20250905 07:53:29.676476 6134 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:29.676539 6134 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:29.676649 6134 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:29.677665 6134 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } }
I20250905 07:53:29.678047 6134 leader_election.cc:304] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b6a77ef2a19499189ac3e3aede2789c; no voters:
I20250905 07:53:29.678375 6134 leader_election.cc:290] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:29.678462 6139 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:29.678646 6139 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [term 1 LEADER]: Becoming Leader. State: Replica: 1b6a77ef2a19499189ac3e3aede2789c, State: Running, Role: LEADER
I20250905 07:53:29.679023 6139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } }
I20250905 07:53:29.679644 6134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:29.680260 6139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6a77ef2a19499189ac3e3aede2789c. Latest consensus state: current_term: 1 leader_uuid: "1b6a77ef2a19499189ac3e3aede2789c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } } }
I20250905 07:53:29.680944 6139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:29.680838 6140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b6a77ef2a19499189ac3e3aede2789c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6a77ef2a19499189ac3e3aede2789c" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43081 } } }
I20250905 07:53:29.681241 6140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:29.682574 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:29.669167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.4.49.62:43081)
0905 07:53:29.669608 (+ 441us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:29.669619 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:29.669663 (+ 44us) server_negotiation.cc:408] Connection header received
0905 07:53:29.670229 (+ 566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:29.670248 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:29.670600 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:29.670978 (+ 378us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:29.671898 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.672769 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:29.673644 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.673877 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:29.676410 (+ 2533us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:29.676436 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:29.676449 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:29.676500 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:29.679100 (+ 2600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.679564 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.679571 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.679578 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.679664 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.679943 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.679947 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.679949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.680434 (+ 485us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:29.681201 (+ 767us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:29.681389 (+ 188us) server_negotiation.cc:300] Negotiation successful
0905 07:53:29.681668 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
W20250905 07:53:29.684569 6154 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:29.684701 6154 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:29.684787 6148 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:29.685492 6148 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:29.690752 6148 catalog_manager.cc:1349] Generated new cluster ID: e3a49057337e4b538aedd9add2a1ec9d
I20250905 07:53:29.690832 6148 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:29.715587 6148 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:29.716681 6148 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:29.722790 6148 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Generated new TSK 0
I20250905 07:53:29.723519 6148 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:29.770793 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43081
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:29.912014 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:29.912319 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:29.912397 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:29.915781 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:29.915864 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:29.915923 6162 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250905 07:53:29.915939 6162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:29.915984 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:29.920632 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:29.921747 6162 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:29.922605 6162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:29.929363 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:29.929405 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:29.929363 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:29.929738 6162 server_base.cc:1047] running on GCE node
I20250905 07:53:29.930115 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:29.930691 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:29.931836 6162 hybrid_clock.cc:648] HybridClock initialized: now 1757058809931823 us; error 28 us; skew 500 ppm
Sep 05 07:53:29 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:29.934546 6162 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:29.935669 6162 webserver.cc:480] Webserver started at http://127.4.49.1:42049/ using document root <none> and password file <none>
I20250905 07:53:29.936287 6162 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:29.936342 6162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:29.936529 6162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:29.938401 6162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "4c44097a4f3445059666676649b028c3"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "d6e5fa7038741bc03b4f6ef776544ed8"
server_key_iv: "10b7fe1da2a803ea4736bd861883afb2"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.938889 6162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "4c44097a4f3445059666676649b028c3"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "d6e5fa7038741bc03b4f6ef776544ed8"
server_key_iv: "10b7fe1da2a803ea4736bd861883afb2"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.942589 6162 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:29.945120 6177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:29.946214 6162 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:29.946343 6162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "4c44097a4f3445059666676649b028c3"
format_stamp: "Formatted at 2025-09-05 07:53:29 on dist-test-slave-mnfb"
server_key: "d6e5fa7038741bc03b4f6ef776544ed8"
server_key_iv: "10b7fe1da2a803ea4736bd861883afb2"
server_key_version: "encryptionkey@0"
I20250905 07:53:29.946457 6162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:29.960022 6162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:29.963196 6162 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:29.963402 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:29.965016 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:29.965071 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:29.965112 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:29.965142 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:29 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:29.975273 6162 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:41911
I20250905 07:53:29.975289 6294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:41911 every 8 connection(s)
I20250905 07:53:29.976483 6162 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:29.977497 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:29.965780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:51743 (local address 127.4.49.62:43081)
0905 07:53:29.965932 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:29.965937 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:29.966779 (+ 842us) server_negotiation.cc:408] Connection header received
0905 07:53:29.967871 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:29.967876 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:29.967950 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:29.968081 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:29.969659 (+ 1578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.970218 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:29.971032 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.971221 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:29.973686 (+ 2465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:29.973714 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:29.973719 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:29.973754 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:29.975382 (+ 1628us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.975913 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.975916 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.975917 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.975966 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.976267 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.976269 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.976271 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.976438 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:29.976513 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:29.977154 (+ 641us) server_negotiation.cc:300] Negotiation successful
0905 07:53:29.977304 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250905 07:53:29.977749 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6162
I20250905 07:53:29.977872 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:29.978082 4292 external_mini_cluster.cc:1468] Setting key fccfd05a125e31ea116544dd5c7e64f2
I20250905 07:53:29.978125 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:29.966085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.1:51743)
0905 07:53:29.966605 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:29.966654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:29.967579 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:29.968273 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:29.968285 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:29.968786 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:29.969493 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:29.969509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.970453 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:29.970456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:29.970880 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:29.970887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.971115 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:29.971803 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:29.971828 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:29.973488 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:29.975504 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:29.975509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:29.975520 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:29.975817 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:29.976046 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:29.976071 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:29.976073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:29.976186 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:29.976537 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:29.976543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:29.976836 (+ 293us) client_negotiation.cc:769] Sending connection context
0905 07:53:29.977069 (+ 233us) client_negotiation.cc:240] Negotiation successful
0905 07:53:29.977368 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":109,"threads_started":1}
I20250905 07:53:29.979601 6295 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43081
I20250905 07:53:29.979784 6295 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:29.982023 6295 heartbeater.cc:507] Master 127.4.49.62:43081 requested a full tablet report, sending...
I20250905 07:53:29.983587 6085 ts_manager.cc:194] Registered new tserver with Master: 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911)
I20250905 07:53:29.985106 6085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:51743
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:53:29.988675 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:29.981516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:50259 (local address 127.4.49.62:43081)
0905 07:53:29.981653 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:29.981657 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:29.981926 (+ 269us) server_negotiation.cc:408] Connection header received
0905 07:53:29.982054 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:29.982057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:29.982105 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:29.982177 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:29.982861 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.983586 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:29.984449 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.984660 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:29.985613 (+ 953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:29.985634 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:29.985637 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:29.985667 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:29.987194 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.987648 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.987651 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.987652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.987695 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:29.987904 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:29.987907 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:29.987909 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:29.988047 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:29.988175 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:29.988405 (+ 230us) server_negotiation.cc:300] Negotiation successful
0905 07:53:29.988512 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250905 07:53:29.988695 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:29.981596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.1:50259)
0905 07:53:29.981851 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:29.981866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:29.981970 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:29.982216 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:29.982220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:29.982436 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:29.982709 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:29.982718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.983725 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:29.983728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:29.984310 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:29.984321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:29.984464 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:29.985080 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:29.985096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:29.985481 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:29.987318 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:29.987322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:29.987324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:29.987532 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:29.987769 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:29.987772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:29.987775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:29.987823 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:29.988170 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:29.988173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:29.988259 (+ 86us) client_negotiation.cc:769] Sending connection context
0905 07:53:29.988346 (+ 87us) client_negotiation.cc:240] Negotiation successful
0905 07:53:29.988507 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:30.022852 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43081
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:30.127578 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:30.127815 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:30.127895 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:30.131250 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:30.131330 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:30.131389 6306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250905 07:53:30.131418 6306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:30.131479 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:30.135861 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:30.136994 6306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:30.137897 6306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:30.144254 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:30.144246 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:30.144250 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:30.144505 6306 server_base.cc:1047] running on GCE node
I20250905 07:53:30.144897 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:30.145449 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:30.146610 6306 hybrid_clock.cc:648] HybridClock initialized: now 1757058810146585 us; error 31 us; skew 500 ppm
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:30.149456 6306 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:30.150547 6306 webserver.cc:480] Webserver started at http://127.4.49.2:40547/ using document root <none> and password file <none>
I20250905 07:53:30.151110 6306 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:30.151161 6306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:30.151357 6306 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:30.153142 6306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "97daab570ecd4ba0aa4020824193adb8"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "921ebea64900015bd3eddabecf0005d0"
server_key_iv: "44b5b408b133231e903a27c05c16ea57"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.153595 6306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "97daab570ecd4ba0aa4020824193adb8"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "921ebea64900015bd3eddabecf0005d0"
server_key_iv: "44b5b408b133231e903a27c05c16ea57"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.156961 6306 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:53:30.159149 6321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:30.160203 6306 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 07:53:30.160311 6306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "97daab570ecd4ba0aa4020824193adb8"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "921ebea64900015bd3eddabecf0005d0"
server_key_iv: "44b5b408b133231e903a27c05c16ea57"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.160403 6306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:30.186980 6306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:30.189682 6306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:30.189862 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:30.191321 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:30.191378 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:30.191419 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:30.191435 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:30.201651 6306 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:41257
I20250905 07:53:30.202708 6306 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:30.203083 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.192026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:57041 (local address 127.4.49.62:43081)
0905 07:53:30.192160 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.192164 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.193130 (+ 966us) server_negotiation.cc:408] Connection header received
0905 07:53:30.193997 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.194001 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.194063 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.194134 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.195825 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.196385 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.197082 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.197256 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.199503 (+ 2247us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.199524 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.199527 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.199555 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.201059 (+ 1504us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.201601 (+ 542us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.201605 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.201607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.201672 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.201976 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.201979 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.201981 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.202184 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.202286 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.202784 (+ 498us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.202917 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:53:30.203241 6437 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:41257 every 8 connection(s)
I20250905 07:53:30.203742 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.192452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.2:57041)
0905 07:53:30.192967 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.193011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.193788 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.194316 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.194328 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.194883 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:30.195672 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.195691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.196505 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.196509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.196968 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.196976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.197170 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.197827 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:30.197864 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:30.199350 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:30.201183 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.201189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.201201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.201502 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.201776 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.201780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.201782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.201884 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.202278 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:30.202282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:30.202512 (+ 230us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.202693 (+ 181us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.202947 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20250905 07:53:30.204905 6438 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43081
I20250905 07:53:30.205053 6438 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:30.205549 6438 heartbeater.cc:507] Master 127.4.49.62:43081 requested a full tablet report, sending...
I20250905 07:53:30.206519 6086 ts_manager.cc:194] Registered new tserver with Master: 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257)
I20250905 07:53:30.207171 6086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:57041
I20250905 07:53:30.208547 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6306
I20250905 07:53:30.208644 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:30.208861 4292 external_mini_cluster.cc:1468] Setting key b834948c632a2b71f9c7f094e52a2ffa
I20250905 07:53:30.213274 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.206886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.2:34349)
0905 07:53:30.207109 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.207121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.207202 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.207435 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.207438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.207638 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:30.207886 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.207893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.208865 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.208868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.209333 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.209342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.209443 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.209973 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:30.209988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:30.210304 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:30.211922 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.211926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.211928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.212161 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.212402 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.212404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.212406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.212447 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.212821 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:30.212824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:30.212903 (+ 79us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.212985 (+ 82us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.213115 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20250905 07:53:30.213341 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.206967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:34349 (local address 127.4.49.62:43081)
0905 07:53:30.207082 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.207087 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.207180 (+ 93us) server_negotiation.cc:408] Connection header received
0905 07:53:30.207278 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.207281 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.207323 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.207409 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.208003 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.208749 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.209445 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.209628 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.210425 (+ 797us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.210441 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.210444 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.210466 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.211819 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.212265 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.212269 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.212271 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.212323 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.212509 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.212512 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.212513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.212734 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.212825 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.213055 (+ 230us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.213182 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:30.251096 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43081
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:30.358856 6450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:30.359099 6450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:30.359169 6450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:30.362538 6450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:30.362620 6450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:30.362684 6450 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250905 07:53:30.362715 6450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:53:30.362776 6450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:30.367352 6450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:30.368531 6450 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:30.369426 6450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:30.375897 6455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:30.376127 6450 server_base.cc:1047] running on GCE node
W20250905 07:53:30.375901 6456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:30.375893 6458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:30.376645 6450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:30.377202 6450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:30.378346 6450 hybrid_clock.cc:648] HybridClock initialized: now 1757058810378331 us; error 28 us; skew 500 ppm
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:30.381310 6450 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:30.382460 6450 webserver.cc:480] Webserver started at http://127.4.49.3:35899/ using document root <none> and password file <none>
I20250905 07:53:30.383052 6450 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:30.383105 6450 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:30.383301 6450 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:30.385179 6450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "8f4f1032604c4d43932916dd24f97931"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "0dd7271dd5ebc2645d519a75807b3cbd"
server_key_iv: "f823fec88171ba78e6c47f22e4e60a29"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.385689 6450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "8f4f1032604c4d43932916dd24f97931"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "0dd7271dd5ebc2645d519a75807b3cbd"
server_key_iv: "f823fec88171ba78e6c47f22e4e60a29"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.389272 6450 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:30.391592 6465 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:30.392699 6450 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:30.392819 6450 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "8f4f1032604c4d43932916dd24f97931"
format_stamp: "Formatted at 2025-09-05 07:53:30 on dist-test-slave-mnfb"
server_key: "0dd7271dd5ebc2645d519a75807b3cbd"
server_key_iv: "f823fec88171ba78e6c47f22e4e60a29"
server_key_version: "encryptionkey@0"
I20250905 07:53:30.392917 6450 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:30.411015 6450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:30.414145 6450 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:30.414330 6450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:30.415985 6450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:30.416085 6450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:30.416143 6450 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:30.416167 6450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:30.426939 6450 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:40509
I20250905 07:53:30.426916 6584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:40509 every 8 connection(s)
I20250905 07:53:30.427954 6450 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:30.428124 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.416689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:40285 (local address 127.4.49.62:43081)
0905 07:53:30.416840 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.416845 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.417562 (+ 717us) server_negotiation.cc:408] Connection header received
0905 07:53:30.418484 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.418488 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.418555 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.418663 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.420210 (+ 1547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.420992 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.421794 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.421972 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.424266 (+ 2294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.424290 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.424292 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.424322 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.425975 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.426494 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.426497 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.426498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.426548 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.426980 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.426984 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.426985 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.427187 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.427279 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.427797 (+ 518us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.427924 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:53:30.428845 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.416965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.3:40285)
0905 07:53:30.417408 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.417453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.418239 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.418851 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.418862 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.419349 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:30.419988 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.420002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.421136 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.421139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.421666 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.421675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.421883 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.422584 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:30.422614 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:30.424108 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:30.426115 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.426120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.426131 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.426371 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.426746 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.426750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.426752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.426891 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.427291 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:30.427298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:30.427542 (+ 244us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.427738 (+ 196us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.428019 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20250905 07:53:30.430038 6585 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43081
I20250905 07:53:30.430194 6585 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:30.430727 6585 heartbeater.cc:507] Master 127.4.49.62:43081 requested a full tablet report, sending...
I20250905 07:53:30.431756 6085 ts_manager.cc:194] Registered new tserver with Master: 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:30.432533 6085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:40285
I20250905 07:53:30.437034 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6450
I20250905 07:53:30.437156 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:30.437424 4292 external_mini_cluster.cc:1468] Setting key 27fd0d37ffc1e84e777bb05faa511697
I20250905 07:53:30.439162 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.432100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35273 (local address 127.4.49.62:43081)
0905 07:53:30.432211 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.432214 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.432367 (+ 153us) server_negotiation.cc:408] Connection header received
0905 07:53:30.432473 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.432476 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.432518 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.432581 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.433217 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.433927 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.434555 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.434703 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.435461 (+ 758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.435475 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.435478 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.435493 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.436719 (+ 1226us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.438109 (+ 1390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.438112 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.438113 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.438163 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.438403 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.438405 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.438407 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.438557 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.438642 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.438905 (+ 263us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.439011 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250905 07:53:30.439181 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.431986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.4.49.3:35273)
0905 07:53:30.432281 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.432296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.432399 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.432628 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.432631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.432839 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:30.433114 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.433122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.434052 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.434055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.434445 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.434453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.434547 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.435069 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:30.435082 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:30.435368 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:30.437804 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.437808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.437811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.438019 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.438249 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.438252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.438254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.438320 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.438663 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:30.438667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:30.438754 (+ 87us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.438843 (+ 89us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.438996 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":222,"threads_started":1}
I20250905 07:53:30.439960 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:30.462882 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.455693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.4.49.62:43081)
0905 07:53:30.455821 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.455824 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.455838 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:53:30.455883 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.455886 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.455932 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.456001 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.456876 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.457419 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.458034 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.458154 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.460239 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.460261 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.460263 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.460295 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.461469 (+ 1174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.461843 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.461846 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.461847 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.461895 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.462093 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.462096 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.462098 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.462263 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.462388 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.462604 (+ 216us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.462726 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:53:30.465575 6086 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:30.467907 6086 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:30.481073 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.476425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:40509 (local address 127.0.0.1:56516)
0905 07:53:30.477166 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.477212 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.477427 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.478141 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.478146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.478169 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:30.478491 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.478499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.479626 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.479629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.480571 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.480580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.480676 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.480749 (+ 73us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.480843 (+ 94us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.480942 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":100,"threads_started":1}
I20250905 07:53:30.481693 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.476509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.0.0.1:42322)
0905 07:53:30.477282 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.477293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.477442 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.478243 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.478246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.478259 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:30.478491 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.478499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.479793 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.479796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.480805 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.480816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.480921 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.480939 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.480985 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.481067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":120,"threads_started":1}
I20250905 07:53:30.482172 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.476641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.4.49.1:41911)
0905 07:53:30.477160 (+ 519us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.477166 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.477381 (+ 215us) server_negotiation.cc:408] Connection header received
0905 07:53:30.477608 (+ 227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.477615 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.477820 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.477937 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:30.478627 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.479678 (+ 1051us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.481158 (+ 1480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.481740 (+ 582us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.481873 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.481948 (+ 75us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.482046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":79,"threads_started":1}
I20250905 07:53:30.482273 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.476677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.4.49.3:40509)
0905 07:53:30.477064 (+ 387us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.477070 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.477556 (+ 486us) server_negotiation.cc:408] Connection header received
0905 07:53:30.477631 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.477636 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.477820 (+ 184us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.477940 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:30.478643 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.479497 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.481422 (+ 1925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.481925 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.482010 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.482067 (+ 57us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.482142 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":98,"threads_started":1}
I20250905 07:53:30.482484 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.477184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:41257 (local address 127.0.0.1:40316)
0905 07:53:30.478707 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.478720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.478820 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.479463 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.479466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.479489 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:30.479782 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.479789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.481156 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.481160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.482148 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.482157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.482248 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.482266 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.482312 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.482370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":89,"threads_started":1}
I20250905 07:53:30.483142 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.477616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.4.49.2:41257)
0905 07:53:30.477979 (+ 363us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.477985 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.478980 (+ 995us) server_negotiation.cc:408] Connection header received
0905 07:53:30.479100 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.479105 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.479300 (+ 195us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.479437 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:30.479902 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.481036 (+ 1134us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.482290 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.482808 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.482894 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.482948 (+ 54us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.483012 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20250905 07:53:30.485453 6517 tablet_service.cc:1468] Processing CreateTablet for tablet e878607987134bb9a776991a8cb34a0c (DEFAULT_TABLE table=test-table [id=d713a4c7c5b04222867b453c23d85cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:30.485800 6371 tablet_service.cc:1468] Processing CreateTablet for tablet e878607987134bb9a776991a8cb34a0c (DEFAULT_TABLE table=test-table [id=d713a4c7c5b04222867b453c23d85cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:30.485800 6227 tablet_service.cc:1468] Processing CreateTablet for tablet e878607987134bb9a776991a8cb34a0c (DEFAULT_TABLE table=test-table [id=d713a4c7c5b04222867b453c23d85cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:30.486443 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e878607987134bb9a776991a8cb34a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.486768 6227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e878607987134bb9a776991a8cb34a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.486769 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e878607987134bb9a776991a8cb34a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.492489 6608 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Bootstrap starting.
I20250905 07:53:30.494168 6610 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Bootstrap starting.
I20250905 07:53:30.495478 6608 tablet_bootstrap.cc:654] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.496086 6610 tablet_bootstrap.cc:654] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.496502 6609 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Bootstrap starting.
I20250905 07:53:30.496690 6608 log.cc:826] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:30.497350 6610 log.cc:826] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:30.498602 6608 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: No bootstrap required, opened a new log
I20250905 07:53:30.498814 6608 ts_tablet_manager.cc:1397] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20250905 07:53:30.499181 6609 tablet_bootstrap.cc:654] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.499313 6610 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: No bootstrap required, opened a new log
I20250905 07:53:30.499593 6610 ts_tablet_manager.cc:1397] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250905 07:53:30.500293 6609 log.cc:826] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:30.501703 6608 raft_consensus.cc:357] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.501904 6608 raft_consensus.cc:383] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.501976 6608 raft_consensus.cc:738] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4f1032604c4d43932916dd24f97931, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.502388 6609 tablet_bootstrap.cc:492] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: No bootstrap required, opened a new log
I20250905 07:53:30.502408 6608 consensus_queue.cc:260] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.502625 6609 ts_tablet_manager.cc:1397] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250905 07:53:30.502554 6610 raft_consensus.cc:357] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.502876 6610 raft_consensus.cc:383] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.502977 6610 raft_consensus.cc:738] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.503125 6585 heartbeater.cc:499] Master 127.4.49.62:43081 was elected leader, sending a full tablet report...
I20250905 07:53:30.503505 6610 consensus_queue.cc:260] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.504240 6438 heartbeater.cc:499] Master 127.4.49.62:43081 was elected leader, sending a full tablet report...
I20250905 07:53:30.504595 6610 ts_tablet_manager.cc:1428] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:53:30.504588 6608 ts_tablet_manager.cc:1428] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250905 07:53:30.507714 6609 raft_consensus.cc:357] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.508014 6609 raft_consensus.cc:383] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.508107 6609 raft_consensus.cc:738] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c44097a4f3445059666676649b028c3, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.508562 6609 consensus_queue.cc:260] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.509212 6295 heartbeater.cc:499] Master 127.4.49.62:43081 was elected leader, sending a full tablet report...
I20250905 07:53:30.509398 6609 ts_tablet_manager.cc:1428] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250905 07:53:30.577435 6616 raft_consensus.cc:491] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:30.577636 6616 raft_consensus.cc:513] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.578758 6616 leader_election.cc:290] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509), 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257)
I20250905 07:53:30.582388 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.579135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:40509 (local address 127.4.49.1:46741)
0905 07:53:30.579448 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.579463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.579580 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.579887 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.579890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.579905 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:30.580263 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.580273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.581242 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.581246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.582026 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.582037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.582132 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.582152 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.582186 (+ 34us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.582233 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
I20250905 07:53:30.582628 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.579552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:41257 (local address 127.4.49.1:36571)
0905 07:53:30.579807 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.579820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.579899 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.580329 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.580334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.580360 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:30.580689 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.580697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.581562 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.581565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.582203 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.582214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.582342 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.582361 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.582423 (+ 62us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.582483 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":117,"threads_started":1}
I20250905 07:53:30.582875 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.579161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:46741 (local address 127.4.49.3:40509)
0905 07:53:30.579287 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.579291 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.579528 (+ 237us) server_negotiation.cc:408] Connection header received
0905 07:53:30.579703 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.579707 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.579765 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.579861 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:30.580425 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.581112 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.582151 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.582618 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.582673 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.582727 (+ 54us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.582788 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:53:30.583063 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.579686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:36571 (local address 127.4.49.2:41257)
0905 07:53:30.579847 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.579852 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.579873 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:53:30.579999 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.580002 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.580043 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.580167 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:30.580829 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.581449 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.582337 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.582800 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.582841 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.582897 (+ 56us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.582954 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:53:30.583492 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e878607987134bb9a776991a8cb34a0c" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931" is_pre_election: true
I20250905 07:53:30.583691 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e878607987134bb9a776991a8cb34a0c" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8" is_pre_election: true
I20250905 07:53:30.583806 6538 raft_consensus.cc:2466] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:30.583952 6391 raft_consensus.cc:2466] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:30.584359 6181 leader_election.cc:304] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 8f4f1032604c4d43932916dd24f97931; no voters:
I20250905 07:53:30.584748 6616 raft_consensus.cc:2802] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:30.584838 6616 raft_consensus.cc:491] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:30.584898 6616 raft_consensus.cc:3058] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.585871 6616 raft_consensus.cc:513] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.586215 6616 leader_election.cc:290] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509), 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257)
I20250905 07:53:30.586688 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e878607987134bb9a776991a8cb34a0c" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931"
I20250905 07:53:30.586707 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e878607987134bb9a776991a8cb34a0c" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8"
I20250905 07:53:30.586815 6538 raft_consensus.cc:3058] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.586815 6391 raft_consensus.cc:3058] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.587819 6391 raft_consensus.cc:2466] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:30.587828 6538 raft_consensus.cc:2466] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:30.588217 6181 leader_election.cc:304] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 8f4f1032604c4d43932916dd24f97931; no voters:
I20250905 07:53:30.588425 6616 raft_consensus.cc:2802] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:30.588662 6616 raft_consensus.cc:695] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c44097a4f3445059666676649b028c3, State: Running, Role: LEADER
I20250905 07:53:30.588977 6616 consensus_queue.cc:237] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } }
I20250905 07:53:30.592149 6084 catalog_manager.cc:5582] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c44097a4f3445059666676649b028c3 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } health_report { overall_health: UNKNOWN } } }
Sep 05 07:53:30 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058809, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:30.634001 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.626035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.4.49.62:43081)
0905 07:53:30.626177 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.626180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.626351 (+ 171us) server_negotiation.cc:408] Connection header received
0905 07:53:30.626467 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.626470 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.626517 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.626598 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:30.627418 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.627920 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.628625 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.628757 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.630797 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:30.630819 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:30.630822 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:30.630848 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:30.632244 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.632770 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.632772 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.632774 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.632812 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:30.632979 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:30.632981 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:30.632982 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:30.633133 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:30.633203 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.633651 (+ 448us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.633826 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":55}
I20250905 07:53:30.634006 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.625951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43081 (local address 127.0.0.1:50072)
0905 07:53:30.626263 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.626278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.626388 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.626632 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.626634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.627091 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:30.627309 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.627316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.628076 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.628081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.628492 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.628501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.628631 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.629177 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:30.629191 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:30.630649 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:30.632422 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.632431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.632437 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.632685 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.632876 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:30.632878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:30.632880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:30.632923 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:30.633267 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:30.633276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:30.633427 (+ 151us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.633571 (+ 144us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.633749 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":127,"threads_started":1}
I20250905 07:53:30.638419 6084 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:50072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250905 07:53:30.639166 6084 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:30.644395 6371 tablet_service.cc:1468] Processing CreateTablet for tablet 240da406328a4c968881cec7846711d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c476d2b14714ad8bf6db7ab1af7be73]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:30.644459 6227 tablet_service.cc:1468] Processing CreateTablet for tablet 240da406328a4c968881cec7846711d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c476d2b14714ad8bf6db7ab1af7be73]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:30.644459 6517 tablet_service.cc:1468] Processing CreateTablet for tablet 240da406328a4c968881cec7846711d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c476d2b14714ad8bf6db7ab1af7be73]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:53:30.645088 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240da406328a4c968881cec7846711d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.645087 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240da406328a4c968881cec7846711d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.645087 6227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240da406328a4c968881cec7846711d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.647459 6610 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: Bootstrap starting.
I20250905 07:53:30.647805 6608 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: Bootstrap starting.
I20250905 07:53:30.648663 6610 tablet_bootstrap.cc:654] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.648854 6608 tablet_bootstrap.cc:654] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.649521 6609 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: Bootstrap starting.
I20250905 07:53:30.650234 6610 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: No bootstrap required, opened a new log
I20250905 07:53:30.650234 6608 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: No bootstrap required, opened a new log
I20250905 07:53:30.650346 6610 ts_tablet_manager.cc:1397] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:30.650358 6608 ts_tablet_manager.cc:1397] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:30.650453 6609 tablet_bootstrap.cc:654] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:30.650848 6608 raft_consensus.cc:357] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.650848 6610 raft_consensus.cc:357] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.650990 6610 raft_consensus.cc:383] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.651000 6608 raft_consensus.cc:383] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.651031 6608 raft_consensus.cc:738] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4f1032604c4d43932916dd24f97931, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.651031 6610 raft_consensus.cc:738] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.651163 6608 consensus_queue.cc:260] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.651214 6610 consensus_queue.cc:260] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.651438 6608 ts_tablet_manager.cc:1428] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:30.651755 6615 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.651857 6609 tablet_bootstrap.cc:492] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: No bootstrap required, opened a new log
I20250905 07:53:30.651882 6615 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.651942 6609 ts_tablet_manager.cc:1397] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250905 07:53:30.652304 6614 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.652443 6614 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.652477 6609 raft_consensus.cc:357] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.652581 6609 raft_consensus.cc:383] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:30.652611 6609 raft_consensus.cc:738] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c44097a4f3445059666676649b028c3, State: Initialized, Role: FOLLOWER
I20250905 07:53:30.652750 6609 consensus_queue.cc:260] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.653033 6609 ts_tablet_manager.cc:1428] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:30.653208 6616 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.653330 6616 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.653352 6610 ts_tablet_manager.cc:1428] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250905 07:53:30.678938 6586 tablet.cc:2378] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:53:30.704152 6440 tablet.cc:2378] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:53:30.728385 6296 tablet.cc:2378] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:30.851105 6616 raft_consensus.cc:491] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:30.851266 6616 raft_consensus.cc:513] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.851689 6616 leader_election.cc:290] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:30.852214 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "240da406328a4c968881cec7846711d6" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8" is_pre_election: true
I20250905 07:53:30.852298 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "240da406328a4c968881cec7846711d6" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931" is_pre_election: true
I20250905 07:53:30.852355 6391 raft_consensus.cc:2466] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:30.852465 6538 raft_consensus.cc:2466] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:30.852666 6181 leader_election.cc:304] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:30.852851 6616 raft_consensus.cc:2802] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:30.852909 6616 raft_consensus.cc:491] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:30.852936 6616 raft_consensus.cc:3058] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.853838 6616 raft_consensus.cc:513] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.854153 6616 leader_election.cc:290] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Requested vote from peers 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:30.854548 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "240da406328a4c968881cec7846711d6" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8"
I20250905 07:53:30.854607 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "240da406328a4c968881cec7846711d6" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931"
I20250905 07:53:30.854663 6391 raft_consensus.cc:3058] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.854743 6538 raft_consensus.cc:3058] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:30.855538 6391 raft_consensus.cc:2466] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:30.855629 6538 raft_consensus.cc:2466] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:30.855851 6181 leader_election.cc:304] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:30.856021 6616 raft_consensus.cc:2802] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:30.856144 6616 raft_consensus.cc:695] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c44097a4f3445059666676649b028c3, State: Running, Role: LEADER
I20250905 07:53:30.856287 6616 consensus_queue.cc:237] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:30.857015 6619 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: TxnStatusTablet state changed. Reason: New leader 4c44097a4f3445059666676649b028c3. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.857128 6619 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:30.857415 6631 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 07:53:30.857615 6631 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250905 07:53:30.858135 6084 catalog_manager.cc:5582] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c44097a4f3445059666676649b028c3 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:30.894798 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.891572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.0.0.1:42326)
0905 07:53:30.891737 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.891753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.891878 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.892207 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.892210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.892218 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:30.892421 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.892428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.893144 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.893147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.893556 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.893563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.893672 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.893727 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:30.894494 (+ 767us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:30.894512 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.894573 (+ 61us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.894665 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20250905 07:53:30.894798 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.891734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.4.49.1:41911)
0905 07:53:30.891886 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.891890 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.891905 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:53:30.892008 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.892011 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.892101 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.892177 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:30.892528 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.893034 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.893663 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.893871 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.893931 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:30.894395 (+ 464us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:30.894502 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.894594 (+ 92us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.894655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250905 07:53:30.900486 6538 raft_consensus.cc:1273] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:30.900486 6391 raft_consensus.cc:1273] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:30.901191 6616 consensus_queue.cc:1035] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:30.901386 6619 consensus_queue.cc:1035] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:30.905472 6615 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: TxnStatusTablet state changed. Reason: New leader 4c44097a4f3445059666676649b028c3. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.905472 6614 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: TxnStatusTablet state changed. Reason: New leader 4c44097a4f3445059666676649b028c3. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.905602 6615 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.905602 6614 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.909509 6619 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.909650 6619 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:30.912098 6614 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.912217 6614 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 8f4f1032604c4d43932916dd24f97931: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.912892 6635 mvcc.cc:204] Tried to move back new op lower bound from 7196912889440985088 to 7196912889268830208. Current Snapshot: MvccSnapshot[applied={T|T < 7196912889440985088}]
I20250905 07:53:30.915244 6619 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.915359 6619 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 4c44097a4f3445059666676649b028c3: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:53:30.916728 6615 tablet_replica.cc:440] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } } }
I20250905 07:53:30.916841 6615 tablet_replica.cc:443] T 240da406328a4c968881cec7846711d6 P 97daab570ecd4ba0aa4020824193adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:53:30.921281 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.918804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.4.49.1:41911)
0905 07:53:30.918912 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.918915 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.918929 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:53:30.919041 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.919045 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.919092 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.919153 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:30.919515 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.919997 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.920654 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.920818 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.920861 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:30.920931 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:30.920991 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.921105 (+ 114us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.921154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250905 07:53:30.931924 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.928630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.4.49.1:36379)
0905 07:53:30.928986 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:30.928998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:30.929092 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:30.929333 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:30.929336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:30.929345 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:30.929671 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.929680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.930677 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.930680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:30.931153 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:30.931161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.931265 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.931309 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:30.931671 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:30.931674 (+ 3us) client_negotiation.cc:769] Sending connection context
0905 07:53:30.931716 (+ 42us) client_negotiation.cc:240] Negotiation successful
0905 07:53:30.931775 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20250905 07:53:30.931926 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:30.928710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:36379 (local address 127.4.49.1:41911)
0905 07:53:30.928835 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:30.928840 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:30.929055 (+ 215us) server_negotiation.cc:408] Connection header received
0905 07:53:30.929173 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:30.929176 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:30.929230 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:30.929311 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:30.929767 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.930526 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:30.931257 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:30.931448 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:30.931494 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:30.931580 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:30.931658 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:30.931740 (+ 82us) server_negotiation.cc:300] Negotiation successful
0905 07:53:30.931788 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250905 07:53:30.939090 6391 raft_consensus.cc:1273] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:30.939086 6538 raft_consensus.cc:1273] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:30.939555 6642 consensus_queue.cc:1035] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:30.939718 6619 consensus_queue.cc:1035] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:30.944461 6634 mvcc.cc:204] Tried to move back new op lower bound from 7196912889603010560 to 7196912888175009792. Current Snapshot: MvccSnapshot[applied={T|T < 7196912889603010560}]
I20250905 07:53:30.986166 6084 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:30.986327 6084 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:30.988677 6084 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Sending DeleteTablet for 3 replicas of tablet e878607987134bb9a776991a8cb34a0c
I20250905 07:53:30.989382 6517 tablet_service.cc:1515] Processing DeleteTablet for tablet e878607987134bb9a776991a8cb34a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:30 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:56516
I20250905 07:53:30.989462 6371 tablet_service.cc:1515] Processing DeleteTablet for tablet e878607987134bb9a776991a8cb34a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:30 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:40316
I20250905 07:53:30.989970 6651 tablet_replica.cc:331] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: stopping tablet replica
I20250905 07:53:30.990223 6651 raft_consensus.cc:2241] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:30.990444 6651 raft_consensus.cc:2270] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:30.990561 6227 tablet_service.cc:1515] Processing DeleteTablet for tablet e878607987134bb9a776991a8cb34a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:30 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:42322
I20250905 07:53:30.990952 6652 tablet_replica.cc:331] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: stopping tablet replica
I20250905 07:53:30.991181 6652 raft_consensus.cc:2241] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:30.991289 6084 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:53:30.991529 6652 raft_consensus.cc:2270] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 07:53:30.991742 6084 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:30.993892 6651 ts_tablet_manager.cc:1905] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:30.997241 6371 tablet_service.cc:1468] Processing CreateTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 (DEFAULT_TABLE table=test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:30.997572 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6478b55cd3e4f00a4b153f9aefe9a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:30.997670 6651 ts_tablet_manager.cc:1918] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:30.997803 6650 tablet_replica.cc:331] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: stopping tablet replica
I20250905 07:53:30.997833 6651 log.cc:1199] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/e878607987134bb9a776991a8cb34a0c
I20250905 07:53:30.998098 6650 raft_consensus.cc:2241] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:30.998292 6651 ts_tablet_manager.cc:1939] T e878607987134bb9a776991a8cb34a0c P 97daab570ecd4ba0aa4020824193adb8: Deleting consensus metadata
I20250905 07:53:30.998322 6650 raft_consensus.cc:2270] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:30.999604 6610 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Bootstrap starting.
I20250905 07:53:30.999761 6073 catalog_manager.cc:4928] TS 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257): tablet e878607987134bb9a776991a8cb34a0c (table test-table [id=d713a4c7c5b04222867b453c23d85cde]) successfully deleted
I20250905 07:53:30.999779 6517 tablet_service.cc:1468] Processing CreateTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 (DEFAULT_TABLE table=test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:31.000022 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6478b55cd3e4f00a4b153f9aefe9a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:31.000638 6610 tablet_bootstrap.cc:654] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:31.001348 6650 ts_tablet_manager.cc:1905] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:31.002419 6608 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Bootstrap starting.
I20250905 07:53:31.003376 6608 tablet_bootstrap.cc:654] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:31.003623 6227 tablet_service.cc:1468] Processing CreateTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 (DEFAULT_TABLE table=test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:31.003895 6227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6478b55cd3e4f00a4b153f9aefe9a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:31.005781 6609 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Bootstrap starting.
I20250905 07:53:31.006724 6652 ts_tablet_manager.cc:1905] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:31.006791 6609 tablet_bootstrap.cc:654] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:31.006860 6650 ts_tablet_manager.cc:1918] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:31.006935 6650 log.cc:1199] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/e878607987134bb9a776991a8cb34a0c
I20250905 07:53:31.007234 6650 ts_tablet_manager.cc:1939] T e878607987134bb9a776991a8cb34a0c P 8f4f1032604c4d43932916dd24f97931: Deleting consensus metadata
I20250905 07:53:31.008436 6073 catalog_manager.cc:4928] TS 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509): tablet e878607987134bb9a776991a8cb34a0c (table test-table [id=d713a4c7c5b04222867b453c23d85cde]) successfully deleted
I20250905 07:53:31.009472 6652 ts_tablet_manager.cc:1918] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:31.009564 6652 log.cc:1199] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/e878607987134bb9a776991a8cb34a0c
I20250905 07:53:31.009708 6609 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: No bootstrap required, opened a new log
I20250905 07:53:31.009795 6609 ts_tablet_manager.cc:1397] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:53:31.009835 6652 ts_tablet_manager.cc:1939] T e878607987134bb9a776991a8cb34a0c P 4c44097a4f3445059666676649b028c3: Deleting consensus metadata
I20250905 07:53:31.010331 6609 raft_consensus.cc:357] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.010442 6609 raft_consensus.cc:383] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:31.010476 6609 raft_consensus.cc:738] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c44097a4f3445059666676649b028c3, State: Initialized, Role: FOLLOWER
I20250905 07:53:31.010608 6609 consensus_queue.cc:260] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.010746 6071 catalog_manager.cc:4928] TS 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911): tablet e878607987134bb9a776991a8cb34a0c (table test-table [id=d713a4c7c5b04222867b453c23d85cde]) successfully deleted
I20250905 07:53:31.011410 6608 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: No bootstrap required, opened a new log
I20250905 07:53:31.011518 6608 ts_tablet_manager.cc:1397] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250905 07:53:31.012034 6608 raft_consensus.cc:357] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.012198 6608 raft_consensus.cc:383] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:31.012243 6608 raft_consensus.cc:738] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4f1032604c4d43932916dd24f97931, State: Initialized, Role: FOLLOWER
I20250905 07:53:31.012384 6608 consensus_queue.cc:260] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.012650 6610 tablet_bootstrap.cc:492] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: No bootstrap required, opened a new log
I20250905 07:53:31.012749 6610 ts_tablet_manager.cc:1397] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250905 07:53:31.012771 6608 ts_tablet_manager.cc:1428] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250905 07:53:31.013199 6610 raft_consensus.cc:357] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.013309 6610 raft_consensus.cc:383] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:31.013345 6610 raft_consensus.cc:738] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Initialized, Role: FOLLOWER
I20250905 07:53:31.013463 6610 consensus_queue.cc:260] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.013921 6610 ts_tablet_manager.cc:1428] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:31.014658 6609 ts_tablet_manager.cc:1428] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250905 07:53:31.158133 6616 raft_consensus.cc:491] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:31.158313 6616 raft_consensus.cc:513] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.158839 6616 leader_election.cc:290] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:31.159444 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6478b55cd3e4f00a4b153f9aefe9a00" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931" is_pre_election: true
I20250905 07:53:31.159479 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6478b55cd3e4f00a4b153f9aefe9a00" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8" is_pre_election: true
I20250905 07:53:31.159633 6538 raft_consensus.cc:2466] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:31.159653 6391 raft_consensus.cc:2466] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 0.
I20250905 07:53:31.159992 6181 leader_election.cc:304] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 8f4f1032604c4d43932916dd24f97931; no voters:
I20250905 07:53:31.160296 6616 raft_consensus.cc:2802] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:31.160358 6616 raft_consensus.cc:491] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:31.160396 6616 raft_consensus.cc:3058] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:31.161316 6616 raft_consensus.cc:513] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.161702 6616 leader_election.cc:290] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Requested vote from peers 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:31.162200 6391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6478b55cd3e4f00a4b153f9aefe9a00" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97daab570ecd4ba0aa4020824193adb8"
I20250905 07:53:31.162213 6538 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6478b55cd3e4f00a4b153f9aefe9a00" candidate_uuid: "4c44097a4f3445059666676649b028c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931"
I20250905 07:53:31.162329 6391 raft_consensus.cc:3058] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:31.162351 6538 raft_consensus.cc:3058] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:31.163262 6391 raft_consensus.cc:2466] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:31.163580 6538 raft_consensus.cc:2466] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c44097a4f3445059666676649b028c3 in term 1.
I20250905 07:53:31.163926 6181 leader_election.cc:304] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:31.164223 6616 raft_consensus.cc:2802] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:31.164328 6616 raft_consensus.cc:695] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c44097a4f3445059666676649b028c3, State: Running, Role: LEADER
I20250905 07:53:31.164484 6616 consensus_queue.cc:237] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:31.166399 6086 catalog_manager.cc:5582] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c44097a4f3445059666676649b028c3 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "4c44097a4f3445059666676649b028c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:31.216941 6537 raft_consensus.cc:1273] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:31.216941 6391 raft_consensus.cc:1273] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Refusing update from remote peer 4c44097a4f3445059666676649b028c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:31.217445 6619 consensus_queue.cc:1035] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:31.217907 6642 consensus_queue.cc:1035] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:53:31.232527 6634 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250905 07:53:32.783475 6086 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:32.783634 6086 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:32.786336 6086 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Sending DeleteTablet for 3 replicas of tablet c6478b55cd3e4f00a4b153f9aefe9a00
I20250905 07:53:32.787029 6371 tablet_service.cc:1515] Processing DeleteTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:32 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:40316
I20250905 07:53:32.787036 6517 tablet_service.cc:1515] Processing DeleteTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:32 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:56516
I20250905 07:53:32.787366 6227 tablet_service.cc:1515] Processing DeleteTablet for tablet c6478b55cd3e4f00a4b153f9aefe9a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:32 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:42322
I20250905 07:53:32.787422 6683 tablet_replica.cc:331] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: stopping tablet replica
I20250905 07:53:32.787523 6683 raft_consensus.cc:2241] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:32.787631 6683 raft_consensus.cc:2270] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:32.787711 6684 tablet_replica.cc:331] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: stopping tablet replica
I20250905 07:53:32.787820 6684 raft_consensus.cc:2241] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:32.788023 6684 raft_consensus.cc:2270] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:32.788412 6683 ts_tablet_manager.cc:1905] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:32.788666 6684 ts_tablet_manager.cc:1905] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:32.791235 6684 ts_tablet_manager.cc:1918] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250905 07:53:32.791326 6684 log.cc:1199] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/c6478b55cd3e4f00a4b153f9aefe9a00
I20250905 07:53:32.791589 6684 ts_tablet_manager.cc:1939] T c6478b55cd3e4f00a4b153f9aefe9a00 P 4c44097a4f3445059666676649b028c3: Deleting consensus metadata
I20250905 07:53:32.792106 6683 ts_tablet_manager.cc:1918] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250905 07:53:32.792192 6683 log.cc:1199] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/c6478b55cd3e4f00a4b153f9aefe9a00
I20250905 07:53:32.792433 6683 ts_tablet_manager.cc:1939] T c6478b55cd3e4f00a4b153f9aefe9a00 P 97daab570ecd4ba0aa4020824193adb8: Deleting consensus metadata
I20250905 07:53:32.792451 6071 catalog_manager.cc:4928] TS 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911): tablet c6478b55cd3e4f00a4b153f9aefe9a00 (table test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]) successfully deleted
I20250905 07:53:32.793088 6073 catalog_manager.cc:4928] TS 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257): tablet c6478b55cd3e4f00a4b153f9aefe9a00 (table test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]) successfully deleted
I20250905 07:53:32.795094 6682 tablet_replica.cc:331] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: stopping tablet replica
I20250905 07:53:32.795229 6682 raft_consensus.cc:2241] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:32.795374 6682 raft_consensus.cc:2270] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:32.796005 6682 ts_tablet_manager.cc:1905] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:32.798300 6682 ts_tablet_manager.cc:1918] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250905 07:53:32.798390 6682 log.cc:1199] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/c6478b55cd3e4f00a4b153f9aefe9a00
I20250905 07:53:32.798599 6682 ts_tablet_manager.cc:1939] T c6478b55cd3e4f00a4b153f9aefe9a00 P 8f4f1032604c4d43932916dd24f97931: Deleting consensus metadata
I20250905 07:53:32.799389 6073 catalog_manager.cc:4928] TS 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509): tablet c6478b55cd3e4f00a4b153f9aefe9a00 (table test-table [id=a18a3695c0314c9fa2cc58c0695fbbb1]) successfully deleted
Sep 05 07:53:32 dist-test-slave-mnfb krb5kdc[6038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 05 07:53:32 dist-test-slave-mnfb krb5kdc[6038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:32.814280 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.806831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.4.49.62:43081)
0905 07:53:32.807103 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:32.807106 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:32.807122 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:53:32.807196 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:32.807199 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:32.807245 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:32.807331 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:32.808170 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.808725 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.809374 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.809519 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.811606 (+ 2087us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:32.811631 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:32.811634 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:32.811665 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:32.813044 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:32.813422 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:32.813424 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:32.813426 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:32.813468 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:32.813616 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:32.813618 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:32.813619 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:32.813778 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:32.813877 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:32.814024 (+ 147us) server_negotiation.cc:300] Negotiation successful
0905 07:53:32.814136 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
I20250905 07:53:32.817051 6086 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:32.817560 6086 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:32.822489 6227 tablet_service.cc:1468] Processing CreateTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b (DEFAULT_TABLE table=test-table [id=ebffda499b77483fa754908fc030cb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:32.822620 6517 tablet_service.cc:1468] Processing CreateTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b (DEFAULT_TABLE table=test-table [id=ebffda499b77483fa754908fc030cb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:32.822620 6371 tablet_service.cc:1468] Processing CreateTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b (DEFAULT_TABLE table=test-table [id=ebffda499b77483fa754908fc030cb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:32.822844 6227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ca6ddd23c429ba886bd08d43a1c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:32.822913 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ca6ddd23c429ba886bd08d43a1c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:32.822913 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ca6ddd23c429ba886bd08d43a1c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:32.825132 6693 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Bootstrap starting.
I20250905 07:53:32.825387 6694 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Bootstrap starting.
I20250905 07:53:32.825831 6695 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Bootstrap starting.
I20250905 07:53:32.826371 6694 tablet_bootstrap.cc:654] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:32.826548 6693 tablet_bootstrap.cc:654] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:32.826766 6695 tablet_bootstrap.cc:654] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:32.827761 6694 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: No bootstrap required, opened a new log
I20250905 07:53:32.827837 6694 ts_tablet_manager.cc:1397] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:32.827893 6695 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: No bootstrap required, opened a new log
I20250905 07:53:32.827982 6695 ts_tablet_manager.cc:1397] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:32.828389 6694 raft_consensus.cc:357] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.828512 6694 raft_consensus.cc:383] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:32.828493 6695 raft_consensus.cc:357] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.828562 6694 raft_consensus.cc:738] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4f1032604c4d43932916dd24f97931, State: Initialized, Role: FOLLOWER
I20250905 07:53:32.828595 6695 raft_consensus.cc:383] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:32.828627 6695 raft_consensus.cc:738] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Initialized, Role: FOLLOWER
I20250905 07:53:32.828692 6693 tablet_bootstrap.cc:492] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: No bootstrap required, opened a new log
I20250905 07:53:32.828691 6694 consensus_queue.cc:260] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.828742 6695 consensus_queue.cc:260] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.828783 6693 ts_tablet_manager.cc:1397] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:53:32.829043 6694 ts_tablet_manager.cc:1428] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 07:53:32.829085 6695 ts_tablet_manager.cc:1428] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:32.829239 6693 raft_consensus.cc:357] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.829353 6693 raft_consensus.cc:383] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:32.829386 6693 raft_consensus.cc:738] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c44097a4f3445059666676649b028c3, State: Initialized, Role: FOLLOWER
I20250905 07:53:32.829517 6693 consensus_queue.cc:260] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.829890 6693 ts_tablet_manager.cc:1428] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:32.931625 6700 raft_consensus.cc:491] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:32.931784 6700 raft_consensus.cc:513] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.932973 6700 leader_election.cc:290] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:32.936762 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.933221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:40509 (local address 127.4.49.2:56249)
0905 07:53:32.933566 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:32.933580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:32.933791 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:32.934341 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:32.934344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:32.934363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:32.934682 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:32.934690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.935496 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.935500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:32.936328 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:32.936337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.936453 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.936477 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:53:32.936523 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:53:32.936584 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":193536,"thread_start_us":101,"threads_started":1}
I20250905 07:53:32.936895 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.933202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.4.49.2:41385)
0905 07:53:32.933541 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:32.933557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:32.933796 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:32.934441 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:32.934444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:32.934461 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:32.934693 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:32.934699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.935733 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.935737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:32.936544 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:32.936551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.936636 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.936650 (+ 14us) client_negotiation.cc:769] Sending connection context
0905 07:53:32.936691 (+ 41us) client_negotiation.cc:240] Negotiation successful
0905 07:53:32.936737 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"spinlock_wait_cycles":20096,"thread_start_us":115,"threads_started":1}
I20250905 07:53:32.937633 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.933579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:56249 (local address 127.4.49.3:40509)
0905 07:53:32.933995 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:32.934000 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:32.934014 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:53:32.934131 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:32.934134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:32.934182 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:32.934275 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:32.934842 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.935386 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.936464 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.937304 (+ 840us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.937373 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:32.937430 (+ 57us) server_negotiation.cc:300] Negotiation successful
0905 07:53:32.937490 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20250905 07:53:32.937726 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.933845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:41385 (local address 127.4.49.1:41911)
0905 07:53:32.934146 (+ 301us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:32.934151 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:32.934166 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:53:32.934217 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:32.934221 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:32.934269 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:32.934416 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:32.934803 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.935624 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.937083 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.937496 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.937528 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:32.937575 (+ 47us) server_negotiation.cc:300] Negotiation successful
0905 07:53:32.937619 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20250905 07:53:32.938053 6537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "686ca6ddd23c429ba886bd08d43a1c2b" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931" is_pre_election: true
I20250905 07:53:32.938218 6537 raft_consensus.cc:2466] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 0.
I20250905 07:53:32.938320 6249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "686ca6ddd23c429ba886bd08d43a1c2b" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c44097a4f3445059666676649b028c3" is_pre_election: true
I20250905 07:53:32.938555 6249 raft_consensus.cc:2466] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 0.
I20250905 07:53:32.938617 6325 leader_election.cc:304] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4f1032604c4d43932916dd24f97931, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:32.939043 6700 raft_consensus.cc:2802] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:32.939101 6700 raft_consensus.cc:491] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:32.939138 6700 raft_consensus.cc:3058] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:32.940291 6700 raft_consensus.cc:513] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.941176 6537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "686ca6ddd23c429ba886bd08d43a1c2b" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931"
I20250905 07:53:32.941283 6537 raft_consensus.cc:3058] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:32.942379 6537 raft_consensus.cc:2466] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 1.
I20250905 07:53:32.942732 6325 leader_election.cc:304] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4f1032604c4d43932916dd24f97931, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:32.943002 6700 leader_election.cc:290] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:32.943105 6705 raft_consensus.cc:2802] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:32.943208 6705 raft_consensus.cc:695] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 1 LEADER]: Becoming Leader. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Running, Role: LEADER
I20250905 07:53:32.943449 6705 consensus_queue.cc:237] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:32.944933 6249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "686ca6ddd23c429ba886bd08d43a1c2b" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c44097a4f3445059666676649b028c3"
I20250905 07:53:32.945072 6249 raft_consensus.cc:3058] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:32.946254 6249 raft_consensus.cc:2466] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 1.
I20250905 07:53:32.948230 6084 catalog_manager.cc:5582] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2). New cstate: current_term: 1 leader_uuid: "97daab570ecd4ba0aa4020824193adb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:32.993292 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:32.989093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.4.49.2:41257)
0905 07:53:32.989396 (+ 303us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:32.989401 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:32.989415 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:53:32.989505 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:32.989508 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:32.989581 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:32.989689 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:32.990120 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.990984 (+ 864us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:32.991882 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:32.992125 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:32.992195 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:32.992823 (+ 628us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:32.992936 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:32.993025 (+ 89us) server_negotiation.cc:300] Negotiation successful
0905 07:53:32.993089 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20250905 07:53:33.004210 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:33.001597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:39341 (local address 127.4.49.1:41911)
0905 07:53:33.001720 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:33.001724 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:33.001878 (+ 154us) server_negotiation.cc:408] Connection header received
0905 07:53:33.001997 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:33.002000 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:33.002049 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:33.002122 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:33.002459 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.003007 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:33.003645 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.003777 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:33.003816 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:33.003893 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:33.003949 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:33.004022 (+ 73us) server_negotiation.cc:300] Negotiation successful
0905 07:53:33.004092 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250905 07:53:33.004223 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:33.001485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.4.49.2:39341)
0905 07:53:33.001807 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:33.001819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:33.001920 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:33.002135 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:33.002137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:33.002143 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:33.002375 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:33.002381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.003115 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:33.003117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:33.003558 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:33.003565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.003660 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:33.003702 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:33.003961 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:33.003965 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:53:33.004006 (+ 41us) client_negotiation.cc:240] Negotiation successful
0905 07:53:33.004093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20250905 07:53:33.010311 6249 raft_consensus.cc:1273] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Refusing update from remote peer 97daab570ecd4ba0aa4020824193adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:33.010740 6537 raft_consensus.cc:1273] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Refusing update from remote peer 97daab570ecd4ba0aa4020824193adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:33.010835 6705 consensus_queue.cc:1035] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:33.011114 6700 consensus_queue.cc:1035] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:33.014166 6664 mvcc.cc:204] Tried to move back new op lower bound from 7196912898085810176 to 7196912897819525120. Current Snapshot: MvccSnapshot[applied={T|T < 7196912898085810176}]
I20250905 07:53:33.029209 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:33.026494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:55327 (local address 127.4.49.2:41257)
0905 07:53:33.026608 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:33.026612 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:33.026730 (+ 118us) server_negotiation.cc:408] Connection header received
0905 07:53:33.026850 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:33.026852 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:33.026901 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:33.026959 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:33.027345 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.027874 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:33.028524 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.028731 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:33.028784 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:33.028863 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:33.028936 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:33.029026 (+ 90us) server_negotiation.cc:300] Negotiation successful
0905 07:53:33.029077 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250905 07:53:33.029212 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:33.026410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:41257 (local address 127.4.49.1:55327)
0905 07:53:33.026654 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:33.026670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:33.026766 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:33.027032 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:33.027035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:33.027043 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:53:33.027256 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:33.027262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.027954 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:33.027957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:33.028421 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:33.028429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:33.028533 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:33.028565 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:53:33.028943 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:53:33.028947 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:53:33.029011 (+ 64us) client_negotiation.cc:240] Negotiation successful
0905 07:53:33.029090 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20250905 07:53:33.052119 6084 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:33.052305 6084 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:33.054665 6084 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Sending DeleteTablet for 3 replicas of tablet 686ca6ddd23c429ba886bd08d43a1c2b
I20250905 07:53:33.055351 6371 tablet_service.cc:1515] Processing DeleteTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:33 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:40316
I20250905 07:53:33.055364 6517 tablet_service.cc:1515] Processing DeleteTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:33 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:56516
I20250905 07:53:33.055364 6227 tablet_service.cc:1515] Processing DeleteTablet for tablet 686ca6ddd23c429ba886bd08d43a1c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:33 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:42322
I20250905 07:53:33.055555 6682 tablet_replica.cc:331] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: stopping tablet replica
I20250905 07:53:33.055555 6683 tablet_replica.cc:331] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: stopping tablet replica
I20250905 07:53:33.055564 6684 tablet_replica.cc:331] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: stopping tablet replica
I20250905 07:53:33.055667 6683 raft_consensus.cc:2241] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:33.055668 6682 raft_consensus.cc:2241] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:33.055672 6684 raft_consensus.cc:2241] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:33.055783 6682 raft_consensus.cc:2270] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:33.055783 6684 raft_consensus.cc:2270] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:33.055975 6683 raft_consensus.cc:2270] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:33.056646 6683 ts_tablet_manager.cc:1905] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:33.056811 6684 ts_tablet_manager.cc:1905] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:33.058527 6683 ts_tablet_manager.cc:1918] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:33.058595 6683 log.cc:1199] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/686ca6ddd23c429ba886bd08d43a1c2b
I20250905 07:53:33.058795 6683 ts_tablet_manager.cc:1939] T 686ca6ddd23c429ba886bd08d43a1c2b P 97daab570ecd4ba0aa4020824193adb8: Deleting consensus metadata
I20250905 07:53:33.059238 6684 ts_tablet_manager.cc:1918] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:33.059312 6684 log.cc:1199] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/686ca6ddd23c429ba886bd08d43a1c2b
I20250905 07:53:33.059554 6684 ts_tablet_manager.cc:1939] T 686ca6ddd23c429ba886bd08d43a1c2b P 4c44097a4f3445059666676649b028c3: Deleting consensus metadata
I20250905 07:53:33.059563 6073 catalog_manager.cc:4928] TS 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257): tablet 686ca6ddd23c429ba886bd08d43a1c2b (table test-table [id=ebffda499b77483fa754908fc030cb1e]) successfully deleted
I20250905 07:53:33.060223 6071 catalog_manager.cc:4928] TS 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911): tablet 686ca6ddd23c429ba886bd08d43a1c2b (table test-table [id=ebffda499b77483fa754908fc030cb1e]) successfully deleted
I20250905 07:53:33.061004 6084 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:53:33.061103 6682 ts_tablet_manager.cc:1905] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 07:53:33.061486 6084 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:33.063105 6682 ts_tablet_manager.cc:1918] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:53:33.063186 6682 log.cc:1199] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/686ca6ddd23c429ba886bd08d43a1c2b
I20250905 07:53:33.063407 6682 ts_tablet_manager.cc:1939] T 686ca6ddd23c429ba886bd08d43a1c2b P 8f4f1032604c4d43932916dd24f97931: Deleting consensus metadata
I20250905 07:53:33.064139 6073 catalog_manager.cc:4928] TS 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509): tablet 686ca6ddd23c429ba886bd08d43a1c2b (table test-table [id=ebffda499b77483fa754908fc030cb1e]) successfully deleted
I20250905 07:53:33.066777 6227 tablet_service.cc:1468] Processing CreateTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 (DEFAULT_TABLE table=test-table [id=e916c562accc48b6a76f0f8338c94a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:33.066941 6371 tablet_service.cc:1468] Processing CreateTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 (DEFAULT_TABLE table=test-table [id=e916c562accc48b6a76f0f8338c94a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:33.067005 6517 tablet_service.cc:1468] Processing CreateTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 (DEFAULT_TABLE table=test-table [id=e916c562accc48b6a76f0f8338c94a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:33.067096 6227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914f7d5f811f4ec8b92fca5474f79223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:33.067160 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914f7d5f811f4ec8b92fca5474f79223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:33.067288 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914f7d5f811f4ec8b92fca5474f79223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:33.069451 6693 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3: Bootstrap starting.
I20250905 07:53:33.069612 6694 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Bootstrap starting.
I20250905 07:53:33.070429 6693 tablet_bootstrap.cc:654] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:33.070523 6694 tablet_bootstrap.cc:654] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:33.071030 6695 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Bootstrap starting.
I20250905 07:53:33.071653 6693 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3: No bootstrap required, opened a new log
I20250905 07:53:33.071748 6693 ts_tablet_manager.cc:1397] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:33.072023 6695 tablet_bootstrap.cc:654] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:33.072237 6693 raft_consensus.cc:357] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.072346 6693 raft_consensus.cc:383] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:33.072378 6693 raft_consensus.cc:738] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c44097a4f3445059666676649b028c3, State: Initialized, Role: FOLLOWER
I20250905 07:53:33.072495 6693 consensus_queue.cc:260] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.072782 6693 ts_tablet_manager.cc:1428] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:33.073262 6695 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: No bootstrap required, opened a new log
I20250905 07:53:33.073349 6695 ts_tablet_manager.cc:1397] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250905 07:53:33.073630 6694 tablet_bootstrap.cc:492] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: No bootstrap required, opened a new log
I20250905 07:53:33.073720 6694 ts_tablet_manager.cc:1397] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:53:33.073820 6695 raft_consensus.cc:357] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.073916 6695 raft_consensus.cc:383] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:33.073945 6695 raft_consensus.cc:738] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Initialized, Role: FOLLOWER
I20250905 07:53:33.074060 6695 consensus_queue.cc:260] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.074172 6694 raft_consensus.cc:357] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.074256 6694 raft_consensus.cc:383] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:33.074285 6694 raft_consensus.cc:738] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4f1032604c4d43932916dd24f97931, State: Initialized, Role: FOLLOWER
I20250905 07:53:33.074335 6695 ts_tablet_manager.cc:1428] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 07:53:33.074401 6694 consensus_queue.cc:260] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.074642 6694 ts_tablet_manager.cc:1428] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:53:33.117473 6717 raft_consensus.cc:491] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:33.117633 6717 raft_consensus.cc:513] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.118140 6717 leader_election.cc:290] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:33.118547 6249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "914f7d5f811f4ec8b92fca5474f79223" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c44097a4f3445059666676649b028c3" is_pre_election: true
I20250905 07:53:33.118556 6537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "914f7d5f811f4ec8b92fca5474f79223" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931" is_pre_election: true
I20250905 07:53:33.118696 6537 raft_consensus.cc:2466] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 0.
I20250905 07:53:33.118701 6249 raft_consensus.cc:2466] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 0.
I20250905 07:53:33.119001 6323 leader_election.cc:304] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c44097a4f3445059666676649b028c3, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:33.119208 6717 raft_consensus.cc:2802] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:33.119273 6717 raft_consensus.cc:491] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:33.119315 6717 raft_consensus.cc:3058] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:33.120844 6717 raft_consensus.cc:513] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.121253 6717 leader_election.cc:290] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911), 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509)
I20250905 07:53:33.121628 6537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "914f7d5f811f4ec8b92fca5474f79223" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4f1032604c4d43932916dd24f97931"
I20250905 07:53:33.121644 6249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "914f7d5f811f4ec8b92fca5474f79223" candidate_uuid: "97daab570ecd4ba0aa4020824193adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c44097a4f3445059666676649b028c3"
I20250905 07:53:33.121757 6537 raft_consensus.cc:3058] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:33.121764 6249 raft_consensus.cc:3058] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:33.122601 6537 raft_consensus.cc:2466] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 1.
I20250905 07:53:33.122834 6249 raft_consensus.cc:2466] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97daab570ecd4ba0aa4020824193adb8 in term 1.
I20250905 07:53:33.122933 6325 leader_election.cc:304] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4f1032604c4d43932916dd24f97931, 97daab570ecd4ba0aa4020824193adb8; no voters:
I20250905 07:53:33.123107 6717 raft_consensus.cc:2802] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:33.123194 6717 raft_consensus.cc:695] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 1 LEADER]: Becoming Leader. State: Replica: 97daab570ecd4ba0aa4020824193adb8, State: Running, Role: LEADER
I20250905 07:53:33.123330 6717 consensus_queue.cc:237] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } }
I20250905 07:53:33.125130 6085 catalog_manager.cc:5582] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2). New cstate: current_term: 1 leader_uuid: "97daab570ecd4ba0aa4020824193adb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97daab570ecd4ba0aa4020824193adb8" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 41257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:33.152804 6538 raft_consensus.cc:1273] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Refusing update from remote peer 97daab570ecd4ba0aa4020824193adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:33.152837 6249 raft_consensus.cc:1273] T 914f7d5f811f4ec8b92fca5474f79223 P 4c44097a4f3445059666676649b028c3 [term 1 FOLLOWER]: Refusing update from remote peer 97daab570ecd4ba0aa4020824193adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:33.153211 6717 consensus_queue.cc:1035] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4f1032604c4d43932916dd24f97931" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:33.153451 6700 consensus_queue.cc:1035] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c44097a4f3445059666676649b028c3" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:53:33.169317 6664 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250905 07:53:34.723353 6086 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:34.723515 6086 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:34.726045 6086 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b6a77ef2a19499189ac3e3aede2789c: Sending DeleteTablet for 3 replicas of tablet 914f7d5f811f4ec8b92fca5474f79223
I20250905 07:53:34.726693 6371 tablet_service.cc:1515] Processing DeleteTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:34 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:40316
I20250905 07:53:34.726693 6227 tablet_service.cc:1515] Processing DeleteTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:34 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:42322
I20250905 07:53:34.726776 6517 tablet_service.cc:1515] Processing DeleteTablet for tablet 914f7d5f811f4ec8b92fca5474f79223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:34 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:56516
I20250905 07:53:34.727144 6748 tablet_replica.cc:331] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: stopping tablet replica
I20250905 07:53:34.727134 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6162
I20250905 07:53:34.727144 6749 tablet_replica.cc:331] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: stopping tablet replica
I20250905 07:53:34.728827 6749 raft_consensus.cc:2241] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:34.729019 6749 raft_consensus.cc:2270] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:34.729082 6748 raft_consensus.cc:2241] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:34.729313 6748 raft_consensus.cc:2270] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:34.729862 6749 ts_tablet_manager.cc:1905] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:34.729976 6748 ts_tablet_manager.cc:1905] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:34.732436 6748 ts_tablet_manager.cc:1918] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250905 07:53:34.732437 6749 ts_tablet_manager.cc:1918] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250905 07:53:34.732522 6748 log.cc:1199] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/914f7d5f811f4ec8b92fca5474f79223
I20250905 07:53:34.732522 6749 log.cc:1199] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/914f7d5f811f4ec8b92fca5474f79223
I20250905 07:53:34.732810 6748 ts_tablet_manager.cc:1939] T 914f7d5f811f4ec8b92fca5474f79223 P 97daab570ecd4ba0aa4020824193adb8: Deleting consensus metadata
I20250905 07:53:34.732810 6749 ts_tablet_manager.cc:1939] T 914f7d5f811f4ec8b92fca5474f79223 P 8f4f1032604c4d43932916dd24f97931: Deleting consensus metadata
I20250905 07:53:34.733601 6073 catalog_manager.cc:4928] TS 8f4f1032604c4d43932916dd24f97931 (127.4.49.3:40509): tablet 914f7d5f811f4ec8b92fca5474f79223 (table test-table [id=e916c562accc48b6a76f0f8338c94a4e]) successfully deleted
I20250905 07:53:34.733798 6073 catalog_manager.cc:4928] TS 97daab570ecd4ba0aa4020824193adb8 (127.4.49.2:41257): tablet 914f7d5f811f4ec8b92fca5474f79223 (table test-table [id=e916c562accc48b6a76f0f8338c94a4e]) successfully deleted
W20250905 07:53:34.734867 6071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:41911 (error 108)
I20250905 07:53:34.735080 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6306
I20250905 07:53:34.736726 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:34.736268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:41911 (local address 127.0.0.1:42340)
0905 07:53:34.736547 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:34.736618 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
W20250905 07:53:34.736872 6071 catalog_manager.cc:4671] TS 4c44097a4f3445059666676649b028c3 (127.4.49.1:41911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 914f7d5f811f4ec8b92fca5474f79223: Network error: Client connection negotiation failed: client connection to 127.4.49.1:41911: connect: Connection refused (error 111)
I20250905 07:53:34.741637 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6450
I20250905 07:53:34.747066 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6054
2025-09-05T07:53:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7246 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:34 dist-test-slave-mnfb krb5kdc[6754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:34 dist-test-slave-mnfb krb5kdc[6754](info): set up 2 sockets
Sep 05 07:53:34 dist-test-slave-mnfb krb5kdc[6754](info): commencing operation
krb5kdc: starting...
W20250905 07:53:36.594645 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.825s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:36 dist-test-slave-mnfb krb5kdc[6754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:36.727831 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39781
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:36.835805 6770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:36.836099 6770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:36.836179 6770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:36.839577 6770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:36.839668 6770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:36.839697 6770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:36.839721 6770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:36.839745 6770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:36.844321 6770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:36.845472 6770 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:36.846343 6770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:36.852324 6778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:36.852381 6770 server_base.cc:1047] running on GCE node
W20250905 07:53:36.852324 6776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:36.852344 6775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:36.853241 6770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:36.854245 6770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:36.855434 6770 hybrid_clock.cc:648] HybridClock initialized: now 1757058816855418 us; error 35 us; skew 500 ppm
Sep 05 07:53:36 dist-test-slave-mnfb krb5kdc[6754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058816, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:36.858615 6770 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:36.859746 6770 webserver.cc:480] Webserver started at http://127.4.49.62:34173/ using document root <none> and password file <none>
I20250905 07:53:36.860365 6770 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:36.860426 6770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:36.860616 6770 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:36.862433 6770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "6b33049fa0dc4c7797234f4026c1fe31"
format_stamp: "Formatted at 2025-09-05 07:53:36 on dist-test-slave-mnfb"
server_key: "7f167b04650e62f14efd2ab22e2e450d"
server_key_iv: "e6f6fec634a6b011c7be7ec0b2c0c49b"
server_key_version: "encryptionkey@0"
I20250905 07:53:36.862922 6770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "6b33049fa0dc4c7797234f4026c1fe31"
format_stamp: "Formatted at 2025-09-05 07:53:36 on dist-test-slave-mnfb"
server_key: "7f167b04650e62f14efd2ab22e2e450d"
server_key_iv: "e6f6fec634a6b011c7be7ec0b2c0c49b"
server_key_version: "encryptionkey@0"
I20250905 07:53:36.866391 6770 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:53:36.868713 6785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:36.869774 6770 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 07:53:36.869887 6770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "6b33049fa0dc4c7797234f4026c1fe31"
format_stamp: "Formatted at 2025-09-05 07:53:36 on dist-test-slave-mnfb"
server_key: "7f167b04650e62f14efd2ab22e2e450d"
server_key_iv: "e6f6fec634a6b011c7be7ec0b2c0c49b"
server_key_version: "encryptionkey@0"
I20250905 07:53:36.869983 6770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:36.887391 6770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:36.890506 6770 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:36.890690 6770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:36.897969 6839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:39781 every 8 connection(s)
I20250905 07:53:36.897969 6770 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:39781
I20250905 07:53:36.899107 6770 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:36.902077 6840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:36.903546 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6770
I20250905 07:53:36.903681 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:36.903905 4292 external_mini_cluster.cc:1468] Setting key 553c512e4f2448db64d7009804046f27
I20250905 07:53:36.908916 6840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31: Bootstrap starting.
I20250905 07:53:36.911446 6840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31: Neither blocks nor log segments found. Creating new log.
Sep 05 07:53:36 dist-test-slave-mnfb krb5kdc[6754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:36.912323 6840 log.cc:826] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:36.914655 6840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31: No bootstrap required, opened a new log
I20250905 07:53:36.917256 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:36.905199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.4.49.62:39781)
0905 07:53:36.905664 (+ 465us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:36.905677 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:36.905721 (+ 44us) server_negotiation.cc:408] Connection header received
0905 07:53:36.906397 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:36.906425 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:36.906805 (+ 380us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:36.907184 (+ 379us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:36.907932 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:36.909149 (+ 1217us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:36.909825 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:36.910120 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:36.912377 (+ 2257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:36.912400 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:36.912417 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:36.912463 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:36.914520 (+ 2057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:36.914970 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:36.914976 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:36.914983 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:36.915087 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:36.915301 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:36.915304 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:36.915306 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:36.915745 (+ 439us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:36.915922 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:36.916208 (+ 286us) server_negotiation.cc:300] Negotiation successful
0905 07:53:36.916490 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20250905 07:53:36.917879 6840 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } }
I20250905 07:53:36.918100 6840 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:36.918200 6840 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b33049fa0dc4c7797234f4026c1fe31, State: Initialized, Role: FOLLOWER
I20250905 07:53:36.918648 6840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } }
I20250905 07:53:36.918766 6840 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:36.918819 6840 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:36.918931 6840 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:36.919893 6840 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } }
I20250905 07:53:36.920238 6840 leader_election.cc:304] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b33049fa0dc4c7797234f4026c1fe31; no voters:
I20250905 07:53:36.920533 6840 leader_election.cc:290] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:36.920861 6845 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:36.921041 6845 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [term 1 LEADER]: Becoming Leader. State: Replica: 6b33049fa0dc4c7797234f4026c1fe31, State: Running, Role: LEADER
I20250905 07:53:36.921303 6845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } }
I20250905 07:53:36.921599 6840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:36.922677 6847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b33049fa0dc4c7797234f4026c1fe31. Latest consensus state: current_term: 1 leader_uuid: "6b33049fa0dc4c7797234f4026c1fe31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } } }
I20250905 07:53:36.922778 6847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:36.923138 6854 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:36.923990 6846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b33049fa0dc4c7797234f4026c1fe31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33049fa0dc4c7797234f4026c1fe31" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39781 } } }
I20250905 07:53:36.924094 6846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:36.926522 6854 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:36.931917 6854 catalog_manager.cc:1349] Generated new cluster ID: ca730164153344158df70ff421731ecf
I20250905 07:53:36.932006 6854 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:36.951849 6854 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:36.953198 6854 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:36.965569 6854 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6b33049fa0dc4c7797234f4026c1fe31: Generated new TSK 0
I20250905 07:53:36.966501 6854 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:37.014532 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39781
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:37.129204 6868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:37.129510 6868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:37.129581 6868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:37.133107 6868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:37.133193 6868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:37.133283 6868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:37.137833 6868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:37.138967 6868 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:37.139892 6868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:37.146488 6873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:37.146477 6874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:37.146477 6876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:37.146709 6868 server_base.cc:1047] running on GCE node
I20250905 07:53:37.147140 6868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:37.147727 6868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:37.148885 6868 hybrid_clock.cc:648] HybridClock initialized: now 1757058817148868 us; error 34 us; skew 500 ppm
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:37.151670 6868 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:37.153021 6868 webserver.cc:480] Webserver started at http://127.4.49.1:44397/ using document root <none> and password file <none>
I20250905 07:53:37.153604 6868 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:37.153658 6868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:37.153844 6868 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:37.155634 6868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "f254172f4b214f9f9b069440a706261e"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "841aad6ac713970dca94436ce17b8c15"
server_key_iv: "312bbfd2e66a50895ea2f2711d73e505"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.156153 6868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "f254172f4b214f9f9b069440a706261e"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "841aad6ac713970dca94436ce17b8c15"
server_key_iv: "312bbfd2e66a50895ea2f2711d73e505"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.159761 6868 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:37.162308 6883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.163411 6868 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:37.163547 6868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "f254172f4b214f9f9b069440a706261e"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "841aad6ac713970dca94436ce17b8c15"
server_key_iv: "312bbfd2e66a50895ea2f2711d73e505"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.163654 6868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:37.182709 6868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:37.186023 6868 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:37.186221 6868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:37.186795 6868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:37.187695 6868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:37.187744 6868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.187798 6868 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:37.187825 6868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.197085 6868 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:44673
I20250905 07:53:37.197116 6997 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:44673 every 8 connection(s)
I20250905 07:53:37.198132 6868 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:37.200592 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 6868
I20250905 07:53:37.200707 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:37.200927 4292 external_mini_cluster.cc:1468] Setting key ae308740ed39bd27e0be6946cb51a63f
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:53:37.212376 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.200128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41471 (local address 127.4.49.62:39781)
0905 07:53:37.200275 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:37.200280 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:37.201107 (+ 827us) server_negotiation.cc:408] Connection header received
0905 07:53:37.202067 (+ 960us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:37.202070 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:37.202123 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:37.202201 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:37.203589 (+ 1388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.204266 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.205144 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.205307 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.208710 (+ 3403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:37.208736 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:37.208738 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:37.208771 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:37.210394 (+ 1623us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.210916 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.210919 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.210920 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.210962 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.211239 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.211242 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.211244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.211426 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:37.211503 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:37.212112 (+ 609us) server_negotiation.cc:300] Negotiation successful
0905 07:53:37.212242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:53:37.213176 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.200437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39781 (local address 127.4.49.1:41471)
0905 07:53:37.200947 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:37.200993 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:37.201842 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:37.202394 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:37.202406 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:37.202835 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:37.203407 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.203421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.204416 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.204420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:37.204993 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.205004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.205251 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.206543 (+ 1292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:37.206590 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:37.208497 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:37.210548 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.210554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.210567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.210830 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.211050 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.211053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.211055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.211159 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.211533 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:37.211539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:37.211812 (+ 273us) client_negotiation.cc:769] Sending connection context
0905 07:53:37.212072 (+ 260us) client_negotiation.cc:240] Negotiation successful
0905 07:53:37.212379 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":115,"threads_started":1}
I20250905 07:53:37.214498 6998 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39781
I20250905 07:53:37.214816 6998 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:37.215406 6998 heartbeater.cc:507] Master 127.4.49.62:39781 requested a full tablet report, sending...
I20250905 07:53:37.217000 6804 ts_manager.cc:194] Registered new tserver with Master: f254172f4b214f9f9b069440a706261e (127.4.49.1:44673)
I20250905 07:53:37.218206 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:41471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:37.245566 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39781
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:37.351855 7005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:37.352116 7005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:37.352198 7005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:37.355530 7005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:37.355624 7005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:37.355737 7005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:37.360265 7005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:37.361411 7005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:37.362313 7005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:37.368672 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:37.368700 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:37.368700 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:37.368924 7005 server_base.cc:1047] running on GCE node
I20250905 07:53:37.369294 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:37.369835 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:37.370980 7005 hybrid_clock.cc:648] HybridClock initialized: now 1757058817370961 us; error 34 us; skew 500 ppm
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:37.373670 7005 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:37.374809 7005 webserver.cc:480] Webserver started at http://127.4.49.2:34673/ using document root <none> and password file <none>
I20250905 07:53:37.375362 7005 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:37.375422 7005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:37.375607 7005 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:37.377466 7005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "1f1b98bbdf8d4404a438395522ad35fc"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "f05046ba94f6604d44b99d6fb6e6877e"
server_key_iv: "43c09612d14ccad5a16d32e10429dea6"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.377952 7005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "1f1b98bbdf8d4404a438395522ad35fc"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "f05046ba94f6604d44b99d6fb6e6877e"
server_key_iv: "43c09612d14ccad5a16d32e10429dea6"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.381562 7005 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:37.383940 7021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.385074 7005 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:37.385205 7005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "1f1b98bbdf8d4404a438395522ad35fc"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "f05046ba94f6604d44b99d6fb6e6877e"
server_key_iv: "43c09612d14ccad5a16d32e10429dea6"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.385305 7005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:37.404556 7005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:37.408263 7005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:37.408465 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:37.409102 7005 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:37.410086 7005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:37.410144 7005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.410198 7005 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:37.410228 7005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.419349 7005 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:44733
I20250905 07:53:37.419379 7136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:44733 every 8 connection(s)
I20250905 07:53:37.420437 7005 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:37.420848 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7005
I20250905 07:53:37.420954 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:37.421209 4292 external_mini_cluster.cc:1468] Setting key da7a6c90bedc4a676e93b7459cccad54
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:53:37.434818 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.423701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:36707 (local address 127.4.49.62:39781)
0905 07:53:37.423826 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:37.423829 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:37.423843 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:53:37.424726 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:37.424730 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:37.424790 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:37.424881 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:37.426727 (+ 1846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.427340 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.428044 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.428301 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.431183 (+ 2882us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:37.431211 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:37.431214 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:37.431249 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:37.432743 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.433352 (+ 609us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.433357 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.433359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.433426 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.433705 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.433708 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.433710 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.433911 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:37.434003 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:37.434549 (+ 546us) server_negotiation.cc:300] Negotiation successful
0905 07:53:37.434682 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:53:37.435731 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.422859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39781 (local address 127.4.49.2:36707)
0905 07:53:37.423419 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:37.423471 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:37.424452 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:37.425037 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:37.425047 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:37.425625 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:37.426491 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.426515 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.427495 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.427498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:37.427911 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.427920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.428151 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.429228 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:37.429257 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:37.430966 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:37.432904 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.432912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.432928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.433229 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.433515 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.433517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.433519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.433611 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.433996 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:37.434002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:37.434257 (+ 255us) client_negotiation.cc:769] Sending connection context
0905 07:53:37.434503 (+ 246us) client_negotiation.cc:240] Negotiation successful
0905 07:53:37.434848 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20250905 07:53:37.437037 7137 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39781
I20250905 07:53:37.437361 7137 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:37.438145 7137 heartbeater.cc:507] Master 127.4.49.62:39781 requested a full tablet report, sending...
I20250905 07:53:37.439462 6804 ts_manager.cc:194] Registered new tserver with Master: 1f1b98bbdf8d4404a438395522ad35fc (127.4.49.2:44733)
I20250905 07:53:37.440124 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:36707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:37.468225 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39781
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:37.580395 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:37.580658 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:37.580742 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:37.584220 7144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:37.584302 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:37.584419 7144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:37.588997 7144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:37.590181 7144 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:37.591099 7144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:37.597839 7152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:37.597920 7144 server_base.cc:1047] running on GCE node
W20250905 07:53:37.597839 7150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:37.597841 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:37.598551 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:37.599119 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:37.600265 7144 hybrid_clock.cc:648] HybridClock initialized: now 1757058817600239 us; error 50 us; skew 500 ppm
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:37.603216 7144 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:37.604442 7144 webserver.cc:480] Webserver started at http://127.4.49.3:40547/ using document root <none> and password file <none>
I20250905 07:53:37.605023 7144 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:37.605079 7144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:37.605284 7144 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:37.607146 7144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "e5bf42afff764a8fa3c42d8211429b91"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "118cc7a9d2c777c9dbc9aca51107a5e0"
server_key_iv: "6a54856f50c0c03ef2d80b0c47b0abf0"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.607645 7144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "e5bf42afff764a8fa3c42d8211429b91"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "118cc7a9d2c777c9dbc9aca51107a5e0"
server_key_iv: "6a54856f50c0c03ef2d80b0c47b0abf0"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.611382 7144 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:37.613835 7159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.614938 7144 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:53:37.615059 7144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "e5bf42afff764a8fa3c42d8211429b91"
format_stamp: "Formatted at 2025-09-05 07:53:37 on dist-test-slave-mnfb"
server_key: "118cc7a9d2c777c9dbc9aca51107a5e0"
server_key_iv: "6a54856f50c0c03ef2d80b0c47b0abf0"
server_key_version: "encryptionkey@0"
I20250905 07:53:37.615159 7144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:37.630795 7144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:37.633877 7144 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:37.634073 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:37.634653 7144 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:37.635548 7144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:37.635597 7144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.635650 7144 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:37.635680 7144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:37.645359 7144 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:41465
I20250905 07:53:37.645383 7273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:41465 every 8 connection(s)
I20250905 07:53:37.646412 7144 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:37.654204 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7144
I20250905 07:53:37.654318 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:37.654554 4292 external_mini_cluster.cc:1468] Setting key 3ba6ed83f8ed5de3f1e3868f3b2d8fca
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[6754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:37.658766 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.648245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:56521 (local address 127.4.49.62:39781)
0905 07:53:37.648379 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:37.648384 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:37.649166 (+ 782us) server_negotiation.cc:408] Connection header received
0905 07:53:37.649998 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:37.650001 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:37.650054 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:37.650129 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:37.651484 (+ 1355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.652005 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.652687 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.652833 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.655519 (+ 2686us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:37.655537 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:37.655540 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:37.655574 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:37.657020 (+ 1446us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.657531 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.657534 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.657535 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.657575 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:37.657797 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:37.657799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:37.657800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:37.657955 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:37.658023 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:37.658519 (+ 496us) server_negotiation.cc:300] Negotiation successful
0905 07:53:37.658634 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:53:37.659420 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:37.648521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39781 (local address 127.4.49.3:56521)
0905 07:53:37.648997 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:37.649039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:37.649790 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:37.650286 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:37.650294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:37.650756 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:37.651327 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.651341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.652143 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:37.652146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:37.652565 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:37.652573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:37.652761 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:37.653806 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:37.653834 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:37.655333 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:37.657144 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.657150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.657162 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.657447 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.657635 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:37.657647 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:37.657649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:37.657736 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:37.658023 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:37.658028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:37.658279 (+ 251us) client_negotiation.cc:769] Sending connection context
0905 07:53:37.658459 (+ 180us) client_negotiation.cc:240] Negotiation successful
0905 07:53:37.658702 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20250905 07:53:37.660537 7274 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39781
I20250905 07:53:37.660780 7274 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:37.661304 7274 heartbeater.cc:507] Master 127.4.49.62:39781 requested a full tablet report, sending...
I20250905 07:53:37.662472 6804 ts_manager.cc:194] Registered new tserver with Master: e5bf42afff764a8fa3c42d8211429b91 (127.4.49.3:41465)
I20250905 07:53:37.663062 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:56521
I20250905 07:53:37.668856 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250905 07:53:37.680416 6843 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:37.679117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.4.49.62:39781)
0905 07:53:37.679262 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:37.679266 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:37.679335 (+ 69us) server_negotiation.cc:408] Connection header received
0905 07:53:37.679485 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:37.679488 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:37.679538 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:37.679604 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:37.680294 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50764: BlockingRecv error: recv got EOF from 127.0.0.1:50764 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20250905 07:53:37.681716 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6868
I20250905 07:53:37.687088 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7005
I20250905 07:53:37.692121 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7144
I20250905 07:53:37.697515 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 6770
2025-09-05T07:53:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2948 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[7286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[7286](info): set up 2 sockets
Sep 05 07:53:37 dist-test-slave-mnfb krb5kdc[7286](info): commencing operation
krb5kdc: starting...
W20250905 07:53:39.530771 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.813s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:39 dist-test-slave-mnfb krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:39.676626 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45691
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:45691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:39.784747 7302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:39.785001 7302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:39.785056 7302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:39.788383 7302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:39.788466 7302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:39.788488 7302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:39.788506 7302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:39.788523 7302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:39.793627 7302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:45691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:39.794785 7302 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:39.795643 7302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:39.801366 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:39.801366 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:39.801366 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:39.801699 7302 server_base.cc:1047] running on GCE node
I20250905 07:53:39.802189 7302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:39.803115 7302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:39.804278 7302 hybrid_clock.cc:648] HybridClock initialized: now 1757058819804268 us; error 34 us; skew 500 ppm
Sep 05 07:53:39 dist-test-slave-mnfb krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058819, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:39.806798 7302 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:39.807857 7302 webserver.cc:480] Webserver started at http://127.4.49.62:45307/ using document root <none> and password file <none>
I20250905 07:53:39.808431 7302 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:39.808483 7302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:39.808667 7302 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:39.810374 7302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "50758851245f46a09f1cad00b6573a4e"
format_stamp: "Formatted at 2025-09-05 07:53:39 on dist-test-slave-mnfb"
server_key: "1339ef4dbfa4565f751fb413db26dbfb"
server_key_iv: "746f9e7c55e1e9b04425a2915bf182bd"
server_key_version: "encryptionkey@0"
I20250905 07:53:39.810844 7302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "50758851245f46a09f1cad00b6573a4e"
format_stamp: "Formatted at 2025-09-05 07:53:39 on dist-test-slave-mnfb"
server_key: "1339ef4dbfa4565f751fb413db26dbfb"
server_key_iv: "746f9e7c55e1e9b04425a2915bf182bd"
server_key_version: "encryptionkey@0"
I20250905 07:53:39.814253 7302 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:53:39.816507 7318 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:39.817564 7302 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:39.817685 7302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "50758851245f46a09f1cad00b6573a4e"
format_stamp: "Formatted at 2025-09-05 07:53:39 on dist-test-slave-mnfb"
server_key: "1339ef4dbfa4565f751fb413db26dbfb"
server_key_iv: "746f9e7c55e1e9b04425a2915bf182bd"
server_key_version: "encryptionkey@0"
I20250905 07:53:39.817780 7302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:39.833330 7302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:39.839637 7302 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:39.839864 7302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:39.847662 7370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:45691 every 8 connection(s)
I20250905 07:53:39.847669 7302 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:45691
I20250905 07:53:39.848879 7302 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:39.851815 7371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:39.852423 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7302
I20250905 07:53:39.852545 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:39.852809 4292 external_mini_cluster.cc:1468] Setting key 3913c567958e7c755f359e39f10cf1d1
I20250905 07:53:39.858541 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: Bootstrap starting.
Sep 05 07:53:39 dist-test-slave-mnfb krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:39.861173 7371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:39.862056 7371 log.cc:826] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:39.864127 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: No bootstrap required, opened a new log
I20250905 07:53:39.865670 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:39.854189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.4.49.62:45691)
0905 07:53:39.854686 (+ 497us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:39.854698 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:39.854736 (+ 38us) server_negotiation.cc:408] Connection header received
0905 07:53:39.855395 (+ 659us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:39.855425 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:39.855765 (+ 340us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:39.856127 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:39.857090 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:39.857866 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:39.858542 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:39.858825 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:39.861178 (+ 2353us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:39.861196 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:39.861210 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:39.861266 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:39.863089 (+ 1823us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:39.863537 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:39.863543 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:39.863548 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:39.863628 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:39.863858 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:39.863862 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:39.863864 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:39.864296 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:39.864498 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:39.864773 (+ 275us) server_negotiation.cc:300] Negotiation successful
0905 07:53:39.865038 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":171,"threads_started":1}
I20250905 07:53:39.866753 7371 raft_consensus.cc:357] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } }
I20250905 07:53:39.866966 7371 raft_consensus.cc:383] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:39.867031 7371 raft_consensus.cc:738] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50758851245f46a09f1cad00b6573a4e, State: Initialized, Role: FOLLOWER
I20250905 07:53:39.867488 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } }
I20250905 07:53:39.867625 7371 raft_consensus.cc:397] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:39.867707 7371 raft_consensus.cc:491] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:39.867823 7371 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:39.868777 7371 raft_consensus.cc:513] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } }
I20250905 07:53:39.869127 7371 leader_election.cc:304] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50758851245f46a09f1cad00b6573a4e; no voters:
I20250905 07:53:39.869443 7371 leader_election.cc:290] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:39.869562 7376 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:39.869778 7376 raft_consensus.cc:695] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [term 1 LEADER]: Becoming Leader. State: Replica: 50758851245f46a09f1cad00b6573a4e, State: Running, Role: LEADER
I20250905 07:53:39.870452 7376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } }
I20250905 07:53:39.870610 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:39.872293 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50758851245f46a09f1cad00b6573a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } } }
I20250905 07:53:39.872428 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:39.872733 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50758851245f46a09f1cad00b6573a4e. Latest consensus state: current_term: 1 leader_uuid: "50758851245f46a09f1cad00b6573a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758851245f46a09f1cad00b6573a4e" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45691 } } }
I20250905 07:53:39.872995 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e [sys.catalog]: This master's current role is: LEADER
W20250905 07:53:39.875094 7391 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:39.875169 7391 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:39.875242 7390 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:39.875962 7390 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:39.881323 7390 catalog_manager.cc:1349] Generated new cluster ID: 87e848d8e8044d17b3db2406241a7b7a
I20250905 07:53:39.881419 7390 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:39.901620 7390 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:39.903023 7390 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:39.913363 7390 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 50758851245f46a09f1cad00b6573a4e: Generated new TSK 0
I20250905 07:53:39.914351 7390 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:39.974385 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:45691
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:40.105095 7399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:40.105352 7399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:40.105413 7399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:40.108839 7399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:40.108930 7399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:40.109014 7399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:40.113588 7399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:40.114733 7399 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:40.115595 7399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:40.122489 7405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:40.122635 7399 server_base.cc:1047] running on GCE node
W20250905 07:53:40.122538 7407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:40.122545 7404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:40.123129 7399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:40.123726 7399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:40.124886 7399 hybrid_clock.cc:648] HybridClock initialized: now 1757058820124868 us; error 32 us; skew 500 ppm
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:40.127774 7399 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:40.128942 7399 webserver.cc:480] Webserver started at http://127.4.49.1:45869/ using document root <none> and password file <none>
I20250905 07:53:40.129508 7399 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:40.129563 7399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:40.129753 7399 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:40.131608 7399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "c776fe63c34d45eba287c4518efa5db4"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "77256125cfe9dba47a3b8683e96fd857"
server_key_iv: "0991e561e401cf2301b271e3ab114eaa"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.132128 7399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "c776fe63c34d45eba287c4518efa5db4"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "77256125cfe9dba47a3b8683e96fd857"
server_key_iv: "0991e561e401cf2301b271e3ab114eaa"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.135663 7399 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:40.138186 7414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.139287 7399 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:53:40.139426 7399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "c776fe63c34d45eba287c4518efa5db4"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "77256125cfe9dba47a3b8683e96fd857"
server_key_iv: "0991e561e401cf2301b271e3ab114eaa"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.139532 7399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:40.186646 7399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:40.189591 7399 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:40.189786 7399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:40.190357 7399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:40.191252 7399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:40.191305 7399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.191366 7399 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:40.191390 7399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.200176 7399 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:44479
I20250905 07:53:40.200196 7530 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:44479 every 8 connection(s)
I20250905 07:53:40.201247 7399 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:40.201821 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7399
I20250905 07:53:40.201929 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:40.202128 4292 external_mini_cluster.cc:1468] Setting key 5d0f4b0fe5c3f18e5011aca9c345f27d
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:53:40.214691 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.203103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:40861 (local address 127.4.49.62:45691)
0905 07:53:40.203239 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.203243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.204102 (+ 859us) server_negotiation.cc:408] Connection header received
0905 07:53:40.204989 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.204992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.205050 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.205123 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.206489 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.206999 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.207640 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.207827 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.210755 (+ 2928us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:40.210780 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:40.210783 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:40.210815 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:40.212224 (+ 1409us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.212945 (+ 721us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.212949 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.212951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.213019 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.213368 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.213372 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.213374 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.213613 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:40.213712 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:40.214314 (+ 602us) server_negotiation.cc:300] Negotiation successful
0905 07:53:40.214484 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:53:40.215483 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.203444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45691 (local address 127.4.49.1:40861)
0905 07:53:40.203935 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:40.203974 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:40.204777 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:40.205290 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:40.205299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:40.205737 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:40.206323 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.206339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.207126 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.207129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:40.207522 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.207530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.207760 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.208923 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:40.208953 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:40.210556 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:40.212404 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.212411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.212426 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.212832 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.213145 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.213148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.213150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.213269 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.213746 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:40.213753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:40.214025 (+ 272us) client_negotiation.cc:769] Sending connection context
0905 07:53:40.214233 (+ 208us) client_negotiation.cc:240] Negotiation successful
0905 07:53:40.214578 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20250905 07:53:40.216862 7531 heartbeater.cc:344] Connected to a master server at 127.4.49.62:45691
I20250905 07:53:40.217141 7531 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:40.217751 7531 heartbeater.cc:507] Master 127.4.49.62:45691 requested a full tablet report, sending...
I20250905 07:53:40.219409 7335 ts_manager.cc:194] Registered new tserver with Master: c776fe63c34d45eba287c4518efa5db4 (127.4.49.1:44479)
I20250905 07:53:40.220669 7335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:40861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:40.248507 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:45691
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:40.358608 7538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:40.358879 7538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:40.358942 7538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:40.362447 7538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:40.362541 7538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:40.362627 7538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:40.367422 7538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:40.368618 7538 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:40.369493 7538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:40.376703 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:40.376709 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:40.376709 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:40.377059 7538 server_base.cc:1047] running on GCE node
I20250905 07:53:40.377528 7538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:40.378154 7538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:40.379348 7538 hybrid_clock.cc:648] HybridClock initialized: now 1757058820379322 us; error 43 us; skew 500 ppm
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:40.382454 7538 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:40.383668 7538 webserver.cc:480] Webserver started at http://127.4.49.2:36721/ using document root <none> and password file <none>
I20250905 07:53:40.384354 7538 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:40.384416 7538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:40.384616 7538 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:40.386505 7538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "51c4905186ec4a1e8780b07ef5ae6c57"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "851f18ab5db7cd0f9296c272c53751ca"
server_key_iv: "5404ab4a16d8da44ef36f3ac5bb6dec8"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.387017 7538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "51c4905186ec4a1e8780b07ef5ae6c57"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "851f18ab5db7cd0f9296c272c53751ca"
server_key_iv: "5404ab4a16d8da44ef36f3ac5bb6dec8"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.390784 7538 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:40.393332 7553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.394524 7538 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:40.394663 7538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "51c4905186ec4a1e8780b07ef5ae6c57"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "851f18ab5db7cd0f9296c272c53751ca"
server_key_iv: "5404ab4a16d8da44ef36f3ac5bb6dec8"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.394774 7538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:40.412525 7538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:40.416150 7538 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:40.416368 7538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:40.416967 7538 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:40.417919 7538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:40.417973 7538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.418027 7538 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:40.418054 7538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.427022 7538 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:38589
I20250905 07:53:40.427104 7667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:38589 every 8 connection(s)
I20250905 07:53:40.428095 7538 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:40.434576 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7538
I20250905 07:53:40.434708 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:40.434966 4292 external_mini_cluster.cc:1468] Setting key af353281779de725b8bce858ef1d7be0
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:40.441537 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.429948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:53083 (local address 127.4.49.62:45691)
0905 07:53:40.430102 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.430106 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.430816 (+ 710us) server_negotiation.cc:408] Connection header received
0905 07:53:40.431693 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.431698 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.431763 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.431835 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.433217 (+ 1382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.433745 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.434840 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.435047 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.437723 (+ 2676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:40.437747 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:40.437750 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:40.437784 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:40.439256 (+ 1472us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.439924 (+ 668us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.439927 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.439928 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.439980 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.440334 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.440336 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.440337 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.440508 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:40.440584 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:40.441268 (+ 684us) server_negotiation.cc:300] Negotiation successful
0905 07:53:40.441391 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:53:40.442533 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.430215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45691 (local address 127.4.49.2:53083)
0905 07:53:40.430670 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:40.430709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:40.431474 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:40.432017 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:40.432025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:40.432497 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:40.433064 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.433077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.433871 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.433874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:40.434265 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.434272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.434492 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.435704 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:40.435734 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:40.437544 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:40.439408 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.439416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.439432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.439816 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.440094 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.440098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.440100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.440245 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.440613 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:40.440620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:40.440982 (+ 362us) client_negotiation.cc:769] Sending connection context
0905 07:53:40.441250 (+ 268us) client_negotiation.cc:240] Negotiation successful
0905 07:53:40.441596 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":140,"threads_started":1}
I20250905 07:53:40.443670 7668 heartbeater.cc:344] Connected to a master server at 127.4.49.62:45691
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:53:40.443926 7668 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:40.444535 7668 heartbeater.cc:507] Master 127.4.49.62:45691 requested a full tablet report, sending...
I20250905 07:53:40.445569 7335 ts_manager.cc:194] Registered new tserver with Master: 51c4905186ec4a1e8780b07ef5ae6c57 (127.4.49.2:38589)
I20250905 07:53:40.446164 7335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:53083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:40.479801 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:45691
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:40.588469 7675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:40.588750 7675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:40.588835 7675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:40.592398 7675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:40.592483 7675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:40.592590 7675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:40.597239 7675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:40.598429 7675 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:40.599376 7675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:40.606272 7680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:40.606272 7681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:40.606272 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:40.606540 7675 server_base.cc:1047] running on GCE node
I20250905 07:53:40.607023 7675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:40.607637 7675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:40.608803 7675 hybrid_clock.cc:648] HybridClock initialized: now 1757058820608780 us; error 45 us; skew 500 ppm
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:40.611888 7675 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:40.613214 7675 webserver.cc:480] Webserver started at http://127.4.49.3:44921/ using document root <none> and password file <none>
I20250905 07:53:40.613871 7675 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:40.613943 7675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:40.614144 7675 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:40.616184 7675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "de8625d2387640b0af65f2716727a542"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "9969c2f46f2d1b75f6d208a2bd2786df"
server_key_iv: "7eed12b780407a44ec6185177dfda419"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.616703 7675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "de8625d2387640b0af65f2716727a542"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "9969c2f46f2d1b75f6d208a2bd2786df"
server_key_iv: "7eed12b780407a44ec6185177dfda419"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.620383 7675 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:40.622964 7690 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.624289 7675 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:40.624452 7675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "de8625d2387640b0af65f2716727a542"
format_stamp: "Formatted at 2025-09-05 07:53:40 on dist-test-slave-mnfb"
server_key: "9969c2f46f2d1b75f6d208a2bd2786df"
server_key_iv: "7eed12b780407a44ec6185177dfda419"
server_key_version: "encryptionkey@0"
I20250905 07:53:40.624567 7675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:40.654443 7675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:40.657693 7675 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:40.657878 7675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:40.658448 7675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:40.659339 7675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:40.659390 7675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.659453 7675 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:40.659476 7675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:40.668707 7675 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:36763
I20250905 07:53:40.668725 7808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:36763 every 8 connection(s)
I20250905 07:53:40.669770 7675 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:40.676249 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7675
I20250905 07:53:40.676383 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:40.676661 4292 external_mini_cluster.cc:1468] Setting key b343e8de4507315fdcf82288970dacf5
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058820, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:40.682911 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.671523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:42973 (local address 127.4.49.62:45691)
0905 07:53:40.671656 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.671659 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.672436 (+ 777us) server_negotiation.cc:408] Connection header received
0905 07:53:40.673363 (+ 927us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.673366 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.673436 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.673509 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.674864 (+ 1355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.675405 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.676098 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.676294 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.679476 (+ 3182us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:40.679493 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:40.679496 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:40.679525 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:40.680919 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.681437 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.681439 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.681441 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.681492 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:40.681772 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:40.681776 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:40.681778 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:40.681955 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:40.682043 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:40.682592 (+ 549us) server_negotiation.cc:300] Negotiation successful
0905 07:53:40.682737 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:53:40.683722 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:40.671791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45691 (local address 127.4.49.3:42973)
0905 07:53:40.672276 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:40.672319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:40.673152 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:40.673675 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:40.673684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:40.674102 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:40.674690 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.674704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.675529 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:40.675533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:40.675943 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:40.675951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:40.676166 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:40.677347 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:40.677380 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:40.679310 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:40.681048 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.681055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.681071 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.681332 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.681572 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:40.681575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:40.681577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:40.681688 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:40.682060 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:40.682068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:40.682326 (+ 258us) client_negotiation.cc:769] Sending connection context
0905 07:53:40.682540 (+ 214us) client_negotiation.cc:240] Negotiation successful
0905 07:53:40.682860 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":116,"threads_started":1}
I20250905 07:53:40.685084 7809 heartbeater.cc:344] Connected to a master server at 127.4.49.62:45691
I20250905 07:53:40.685350 7809 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:40.685948 7809 heartbeater.cc:507] Master 127.4.49.62:45691 requested a full tablet report, sending...
I20250905 07:53:40.687003 7335 ts_manager.cc:194] Registered new tserver with Master: de8625d2387640b0af65f2716727a542 (127.4.49.3:36763)
I20250905 07:53:40.687585 7335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:42973
I20250905 07:53:40.690999 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250905 07:53:40.826870 7374 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:40.822943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.4.49.62:45691)
0905 07:53:40.823102 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.823106 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.824197 (+ 1091us) server_negotiation.cc:408] Connection header received
0905 07:53:40.825213 (+ 1016us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.825216 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.825271 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.825345 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.826725 (+ 1380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56050: BlockingRecv error: recv got EOF from 127.0.0.1:56050 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20250905 07:53:40.828366 7374 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:40.827563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.4.49.62:45691)
0905 07:53:40.827687 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.827690 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.827703 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:53:40.827798 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.827800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.827843 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.827907 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.828259 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56066: BlockingRecv error: recv got EOF from 127.0.0.1:56066 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20250905 07:53:40.829310 7374 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:40.828674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.4.49.62:45691)
0905 07:53:40.828783 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.828786 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.828798 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:53:40.828830 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.828833 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.828869 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.828932 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.829208 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56076: BlockingRecv error: recv got EOF from 127.0.0.1:56076 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20250905 07:53:40.833585 7374 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:40.832824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.4.49.62:45691)
0905 07:53:40.832949 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:40.832952 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:40.832965 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:53:40.833004 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:40.833006 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:40.833048 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:40.833117 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:40.833487 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56090: BlockingRecv error: recv got EOF from 127.0.0.1:56090 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:53:40.841571 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7399
I20250905 07:53:40.846860 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7538
I20250905 07:53:40.854204 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7675
I20250905 07:53:40.859834 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7302
2025-09-05T07:53:40Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3163 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7832](info): set up 2 sockets
Sep 05 07:53:40 dist-test-slave-mnfb krb5kdc[7832](info): commencing operation
krb5kdc: starting...
W20250905 07:53:42.766530 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.885s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20250905 07:53:43.012585 4292 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.246s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20250905 07:53:43.146222 4292 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.134s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20250905 07:53:43.330914 4292 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.185s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/test-user.keytab.
W20250905 07:53:43.975188 4292 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.644s user 0.000s sys 0.002s
Sep 05 07:53:43 dist-test-slave-mnfb krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab.
W20250905 07:53:45.197467 4292 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.4.49.62: real 1.201s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab.
W20250905 07:53:45.805769 4292 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.4.49.62: real 0.608s user 0.000s sys 0.003s
I20250905 07:53:45.806293 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44487
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:44487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:45.918800 7848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:45.919057 7848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:45.919114 7848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:45.922559 7848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:45.922644 7848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:45.922667 7848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:45.922685 7848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:45.922701 7848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:45.927320 7848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:44487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:45.928504 7848 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:45.929392 7848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:45.935472 7856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:45.935456 7853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:45.935452 7854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:45.935576 7848 server_base.cc:1047] running on GCE node
I20250905 07:53:45.936306 7848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:45.937291 7848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:45.938465 7848 hybrid_clock.cc:648] HybridClock initialized: now 1757058825938455 us; error 36 us; skew 500 ppm
Sep 05 07:53:45 dist-test-slave-mnfb krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:45.941345 7848 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:45.942582 7848 webserver.cc:480] Webserver started at http://127.4.49.62:33699/ using document root <none> and password file <none>
I20250905 07:53:45.943176 7848 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:45.943231 7848 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:45.943426 7848 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:45.945333 7848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "63f4acaa833748fe87fb877c726c7691"
format_stamp: "Formatted at 2025-09-05 07:53:45 on dist-test-slave-mnfb"
server_key: "6b731c397beddb41a3aa0864322d4093"
server_key_iv: "06c8c556451bf9d77fbd07c394f84540"
server_key_version: "encryptionkey@0"
I20250905 07:53:45.945848 7848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "63f4acaa833748fe87fb877c726c7691"
format_stamp: "Formatted at 2025-09-05 07:53:45 on dist-test-slave-mnfb"
server_key: "6b731c397beddb41a3aa0864322d4093"
server_key_iv: "06c8c556451bf9d77fbd07c394f84540"
server_key_version: "encryptionkey@0"
I20250905 07:53:45.949535 7848 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:45.952086 7863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:45.953238 7848 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:45.953363 7848 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "63f4acaa833748fe87fb877c726c7691"
format_stamp: "Formatted at 2025-09-05 07:53:45 on dist-test-slave-mnfb"
server_key: "6b731c397beddb41a3aa0864322d4093"
server_key_iv: "06c8c556451bf9d77fbd07c394f84540"
server_key_version: "encryptionkey@0"
I20250905 07:53:45.953475 7848 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:45.975250 7848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:45.982807 7848 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:45.983006 7848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:45.990689 7848 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:44487
I20250905 07:53:45.990669 7917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:44487 every 8 connection(s)
I20250905 07:53:45.991792 7848 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:45.992650 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7848
I20250905 07:53:45.992750 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:45.992972 4292 external_mini_cluster.cc:1468] Setting key 4159361351c7f16b8980224e18076ab9
I20250905 07:53:45.995126 7918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 05 07:53:45 dist-test-slave-mnfb krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:46.000648 7918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691: Bootstrap starting.
I20250905 07:53:46.003016 7918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:46.003751 7918 log.cc:826] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:46.005369 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:45.994200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.4.49.62:44487)
0905 07:53:45.994672 (+ 472us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:45.994682 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:45.994719 (+ 37us) server_negotiation.cc:408] Connection header received
0905 07:53:45.995380 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:45.995401 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:45.995723 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:45.996034 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:45.997059 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:45.997853 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:45.998501 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:45.998787 (+ 286us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:46.001104 (+ 2317us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:46.001149 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:46.001160 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:46.001197 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:46.002848 (+ 1651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:46.003293 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:46.003297 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:46.003302 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:46.003383 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:46.003614 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:46.003618 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:46.003620 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:46.003995 (+ 375us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:46.004228 (+ 233us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:46.004506 (+ 278us) server_negotiation.cc:300] Negotiation successful
0905 07:53:46.004761 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":158,"threads_started":1}
I20250905 07:53:46.005714 7918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691: No bootstrap required, opened a new log
I20250905 07:53:46.008100 4292 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250905 07:53:46.008387 7918 raft_consensus.cc:357] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } }
I20250905 07:53:46.008600 7918 raft_consensus.cc:383] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:46.008661 7918 raft_consensus.cc:738] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f4acaa833748fe87fb877c726c7691, State: Initialized, Role: FOLLOWER
I20250905 07:53:46.009088 7918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } }
I20250905 07:53:46.009220 7918 raft_consensus.cc:397] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:46.009279 7918 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:46.009387 7918 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:46.010500 7918 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } }
I20250905 07:53:46.010897 7918 leader_election.cc:304] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63f4acaa833748fe87fb877c726c7691; no voters:
I20250905 07:53:46.011229 7918 leader_election.cc:290] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:46.011499 7924 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:46.011749 7924 raft_consensus.cc:695] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [term 1 LEADER]: Becoming Leader. State: Replica: 63f4acaa833748fe87fb877c726c7691, State: Running, Role: LEADER
I20250905 07:53:46.012152 7924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } }
I20250905 07:53:46.012594 7918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:46.014691 7925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63f4acaa833748fe87fb877c726c7691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } } }
I20250905 07:53:46.014827 7925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:46.015242 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63f4acaa833748fe87fb877c726c7691. Latest consensus state: current_term: 1 leader_uuid: "63f4acaa833748fe87fb877c726c7691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f4acaa833748fe87fb877c726c7691" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44487 } } }
I20250905 07:53:46.015419 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:46.015430 7941 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:46.018250 7941 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:46.024971 7941 catalog_manager.cc:1349] Generated new cluster ID: 24af3224e80b4df9b3fb24d802bc718f
I20250905 07:53:46.025050 7941 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:46.025974 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:46.021107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.4.49.62:44487)
0905 07:53:46.021253 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:46.021258 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:46.021487 (+ 229us) server_negotiation.cc:408] Connection header received
0905 07:53:46.021613 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:46.021616 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:46.021666 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:46.021733 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:46.022464 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:46.023209 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:46.024098 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:46.024309 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:46.024848 (+ 539us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:46.024866 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:46.024870 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:46.025448 (+ 578us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:46.025487 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:46.025508 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:46.025603 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:46.025763 (+ 160us) server_negotiation.cc:300] Negotiation successful
0905 07:53:46.025837 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250905 07:53:46.031980 7941 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:46.033326 7941 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:46.048859 7941 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 63f4acaa833748fe87fb877c726c7691: Generated new TSK 0
I20250905 07:53:46.049858 7941 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:53:46.056488 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7848
2025-09-05T07:53:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (5194 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:46 dist-test-slave-mnfb krb5kdc[7951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:46 dist-test-slave-mnfb krb5kdc[7951](info): set up 2 sockets
Sep 05 07:53:46 dist-test-slave-mnfb krb5kdc[7951](info): commencing operation
krb5kdc: starting...
W20250905 07:53:47.894824 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:47 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:48.028241 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36915
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:48.139465 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:48.139743 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:48.139808 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:48.143205 7967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:48.143290 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:48.143312 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:48.143332 7967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:48.143347 7967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:48.147923 7967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:36915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:48.149168 7967 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:48.150094 7967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:48.155836 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.155838 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.155838 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:48.156152 7967 server_base.cc:1047] running on GCE node
I20250905 07:53:48.156684 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:48.157671 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:48.158826 7967 hybrid_clock.cc:648] HybridClock initialized: now 1757058828158814 us; error 31 us; skew 500 ppm
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:48.161541 7967 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:48.162717 7967 webserver.cc:480] Webserver started at http://127.4.49.62:41563/ using document root <none> and password file <none>
I20250905 07:53:48.163298 7967 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:48.163352 7967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:48.163554 7967 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:48.165445 7967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "ae545edd6c314ec2948960aaa0598239"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "acb8535013123a9c90cb43e23b32ef43"
server_key_iv: "75ed3f5c1cb94ca6f84033688c3becc8"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.165957 7967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "ae545edd6c314ec2948960aaa0598239"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "acb8535013123a9c90cb43e23b32ef43"
server_key_iv: "75ed3f5c1cb94ca6f84033688c3becc8"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.169644 7967 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250905 07:53:48.172111 7982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.173462 7967 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:53:48.173606 7967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "ae545edd6c314ec2948960aaa0598239"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "acb8535013123a9c90cb43e23b32ef43"
server_key_iv: "75ed3f5c1cb94ca6f84033688c3becc8"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.173728 7967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:48.206259 7967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:48.209565 7967 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:48.209771 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:48.217314 7967 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:36915
I20250905 07:53:48.217308 8036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:36915 every 8 connection(s)
I20250905 07:53:48.218430 7967 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:48.221426 8037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:48.224298 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 7967
I20250905 07:53:48.224429 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:48.224637 4292 external_mini_cluster.cc:1468] Setting key 8692797a393810b6bae169c81118c569
I20250905 07:53:48.227625 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239: Bootstrap starting.
I20250905 07:53:48.230299 8037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:48.231199 8037 log.cc:826] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239: Log is configured to *not* fsync() on all Append() calls
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:48.233443 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239: No bootstrap required, opened a new log
I20250905 07:53:48.236363 8037 raft_consensus.cc:357] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } }
I20250905 07:53:48.236658 8037 raft_consensus.cc:383] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:48.236765 8037 raft_consensus.cc:738] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae545edd6c314ec2948960aaa0598239, State: Initialized, Role: FOLLOWER
I20250905 07:53:48.237252 8037 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } }
I20250905 07:53:48.237406 8037 raft_consensus.cc:397] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:48.237471 8037 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:48.237587 8037 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:48.237888 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:48.225968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.4.49.62:36915)
0905 07:53:48.226379 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:48.226389 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:48.226432 (+ 43us) server_negotiation.cc:408] Connection header received
0905 07:53:48.226962 (+ 530us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:48.226980 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:48.227284 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:48.227568 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:48.228682 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.229471 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:48.230326 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.230624 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:48.232990 (+ 2366us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:48.233015 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:48.233030 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:48.233078 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:48.234876 (+ 1798us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.235381 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.235386 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.235391 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.235470 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.235745 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.235748 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.235749 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.236199 (+ 450us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:48.236390 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:48.236719 (+ 329us) server_negotiation.cc:300] Negotiation successful
0905 07:53:48.237130 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20250905 07:53:48.238520 8037 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } }
I20250905 07:53:48.238870 8037 leader_election.cc:304] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae545edd6c314ec2948960aaa0598239; no voters:
I20250905 07:53:48.239166 8037 leader_election.cc:290] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:48.239328 8042 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:48.239584 8042 raft_consensus.cc:695] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [term 1 LEADER]: Becoming Leader. State: Replica: ae545edd6c314ec2948960aaa0598239, State: Running, Role: LEADER
I20250905 07:53:48.239930 8042 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } }
I20250905 07:53:48.240377 8037 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:48.241649 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae545edd6c314ec2948960aaa0598239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } } }
I20250905 07:53:48.241779 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:48.242075 8044 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae545edd6c314ec2948960aaa0598239. Latest consensus state: current_term: 1 leader_uuid: "ae545edd6c314ec2948960aaa0598239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae545edd6c314ec2948960aaa0598239" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36915 } } }
I20250905 07:53:48.242146 8051 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:48.242218 8044 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:48.245074 8051 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:48.250675 8051 catalog_manager.cc:1349] Generated new cluster ID: c84b204844684103818839d79d91d10e
I20250905 07:53:48.250767 8051 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:48.258085 8051 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:48.259059 8051 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:48.268999 8051 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ae545edd6c314ec2948960aaa0598239: Generated new TSK 0
I20250905 07:53:48.269804 8051 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:48.324590 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36915
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:48.451416 8065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:48.451675 8065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:48.451736 8065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:48.455085 8065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:48.455173 8065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:48.455256 8065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:48.459792 8065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:48.461042 8065 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:48.461966 8065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:48.469233 8073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.469312 8070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.469460 8071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:48.469666 8065 server_base.cc:1047] running on GCE node
I20250905 07:53:48.470090 8065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:48.470736 8065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:48.471904 8065 hybrid_clock.cc:648] HybridClock initialized: now 1757058828471881 us; error 59 us; skew 500 ppm
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:48.474776 8065 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:48.475934 8065 webserver.cc:480] Webserver started at http://127.4.49.1:44829/ using document root <none> and password file <none>
I20250905 07:53:48.476557 8065 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:48.476614 8065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:48.476812 8065 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:48.478768 8065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "7e387684a3d64da6b066d8e319ae841d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d2450ca97c8748a15db2b273ee6a011e"
server_key_iv: "20227a5bf60b46d99e74cc9f9b1cd284"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.479310 8065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "7e387684a3d64da6b066d8e319ae841d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d2450ca97c8748a15db2b273ee6a011e"
server_key_iv: "20227a5bf60b46d99e74cc9f9b1cd284"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.483122 8065 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:48.485773 8080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.486903 8065 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:48.487033 8065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "7e387684a3d64da6b066d8e319ae841d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d2450ca97c8748a15db2b273ee6a011e"
server_key_iv: "20227a5bf60b46d99e74cc9f9b1cd284"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.487149 8065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:48.531358 8065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:48.534763 8065 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:48.534962 8065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:48.535548 8065 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:48.536512 8065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:48.536562 8065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.536616 8065 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:48.536643 8065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.546057 8065 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:38555
I20250905 07:53:48.546087 8195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:38555 every 8 connection(s)
I20250905 07:53:48.547127 8065 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:48.552107 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8065
I20250905 07:53:48.552218 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:48.552444 4292 external_mini_cluster.cc:1468] Setting key f86f268356ad628b77989859c4402b34
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:48.561025 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:48.549070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:55009 (local address 127.4.49.62:36915)
0905 07:53:48.549214 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:48.549218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:48.550012 (+ 794us) server_negotiation.cc:408] Connection header received
0905 07:53:48.550856 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:48.550859 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:48.550913 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:48.550996 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:48.552937 (+ 1941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.553750 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:48.554509 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.554699 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:48.557180 (+ 2481us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:48.557203 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:48.557205 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:48.557235 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:48.558787 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.559457 (+ 670us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.559460 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.559461 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.559504 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.559822 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.559824 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.559825 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.559982 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:48.560081 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:48.560764 (+ 683us) server_negotiation.cc:300] Negotiation successful
0905 07:53:48.560890 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:53:48.561985 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:48.549357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36915 (local address 127.4.49.1:55009)
0905 07:53:48.549848 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:48.549890 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:48.550665 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:48.551158 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:48.551167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:48.551610 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:48.552727 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:48.552747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.553906 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:48.553910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:48.554374 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:48.554384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.554582 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:48.555256 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:48.555291 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:48.556967 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:48.558933 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:48.558940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:48.558957 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:48.559347 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:48.559583 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:48.559586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:48.559589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:48.559741 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:48.560107 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:48.560114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:48.560472 (+ 358us) client_negotiation.cc:769] Sending connection context
0905 07:53:48.560741 (+ 269us) client_negotiation.cc:240] Negotiation successful
0905 07:53:48.561068 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":106,"threads_started":1}
I20250905 07:53:48.563354 8196 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36915
I20250905 07:53:48.563628 8196 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:48.564337 8196 heartbeater.cc:507] Master 127.4.49.62:36915 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:53:48.566128 7999 ts_manager.cc:194] Registered new tserver with Master: 7e387684a3d64da6b066d8e319ae841d (127.4.49.1:38555)
I20250905 07:53:48.567363 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:55009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:48.601111 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36915
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:48.716014 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:48.716306 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:48.716401 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:48.719923 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:48.720021 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:48.720163 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:48.724951 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:48.726177 8203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:48.727133 8203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:48.734194 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:48.734392 8203 server_base.cc:1047] running on GCE node
W20250905 07:53:48.734287 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.734194 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:48.734891 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:48.735489 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:48.736646 8203 hybrid_clock.cc:648] HybridClock initialized: now 1757058828736617 us; error 43 us; skew 500 ppm
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:48.739348 8203 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:48.740533 8203 webserver.cc:480] Webserver started at http://127.4.49.2:33803/ using document root <none> and password file <none>
I20250905 07:53:48.741106 8203 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:48.741156 8203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:48.741436 8203 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:48.743532 8203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "6916d55637364becaa5ceaf91c07d4ff"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "08fa789514293b600782683dbce323a3"
server_key_iv: "58eabe94027641442389bb1119e9ec4e"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.744026 8203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "6916d55637364becaa5ceaf91c07d4ff"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "08fa789514293b600782683dbce323a3"
server_key_iv: "58eabe94027641442389bb1119e9ec4e"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.747777 8203 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:48.750290 8218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.751456 8203 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:48.751576 8203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "6916d55637364becaa5ceaf91c07d4ff"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "08fa789514293b600782683dbce323a3"
server_key_iv: "58eabe94027641442389bb1119e9ec4e"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.751703 8203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:48.778359 8203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:48.781458 8203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:48.781630 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:48.782146 8203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:48.783017 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:48.783066 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.783105 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:48.783126 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.791953 8203 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:45389
I20250905 07:53:48.791988 8332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:45389 every 8 connection(s)
I20250905 07:53:48.793028 8203 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:48.797325 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8203
I20250905 07:53:48.797435 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:48.797663 4292 external_mini_cluster.cc:1468] Setting key 22d052bf3e03114a2da8421796c90989
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:53:48.807233 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:48.794930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:32885 (local address 127.4.49.62:36915)
0905 07:53:48.795086 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:48.795089 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:48.795817 (+ 728us) server_negotiation.cc:408] Connection header received
0905 07:53:48.796740 (+ 923us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:48.796744 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:48.796806 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:48.796878 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:48.798745 (+ 1867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.799244 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:48.799912 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.800128 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:48.802736 (+ 2608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:48.802762 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:48.802765 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:48.802798 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:48.804812 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.805452 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.805456 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.805458 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.805517 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:48.805845 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:48.805849 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:48.805850 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:48.806033 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:48.806209 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:48.806897 (+ 688us) server_negotiation.cc:300] Negotiation successful
0905 07:53:48.807054 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250905 07:53:48.807933 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:48.795197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36915 (local address 127.4.49.2:32885)
0905 07:53:48.795673 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:48.795712 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:48.796513 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:48.797062 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:48.797070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:48.797556 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:48.798584 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:48.798601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.799366 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:48.799370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:48.799784 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:48.799792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:48.800017 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:48.800677 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:48.800706 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:48.802516 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:48.804987 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:48.804993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:48.805006 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:48.805322 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:48.805614 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:48.805616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:48.805618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:48.805740 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:48.806208 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:48.806216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:48.806602 (+ 386us) client_negotiation.cc:769] Sending connection context
0905 07:53:48.806840 (+ 238us) client_negotiation.cc:240] Negotiation successful
0905 07:53:48.807176 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20250905 07:53:48.809196 8333 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:53:48.809495 8333 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:48.810248 8333 heartbeater.cc:507] Master 127.4.49.62:36915 requested a full tablet report, sending...
I20250905 07:53:48.811535 7999 ts_manager.cc:194] Registered new tserver with Master: 6916d55637364becaa5ceaf91c07d4ff (127.4.49.2:45389)
I20250905 07:53:48.812315 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:32885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:48.846462 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36915
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:48.957502 8340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:48.957764 8340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:48.957826 8340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:48.961287 8340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:48.961413 8340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:48.961529 8340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:48.966145 8340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:48.967271 8340 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:48.968210 8340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:48.975090 8348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.975095 8346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:48.975093 8345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:48.975523 8340 server_base.cc:1047] running on GCE node
I20250905 07:53:48.975921 8340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:48.976557 8340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:48.977706 8340 hybrid_clock.cc:648] HybridClock initialized: now 1757058828977679 us; error 43 us; skew 500 ppm
Sep 05 07:53:48 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:48.980417 8340 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:48.981565 8340 webserver.cc:480] Webserver started at http://127.4.49.3:46547/ using document root <none> and password file <none>
I20250905 07:53:48.982180 8340 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:48.982235 8340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:48.982431 8340 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:48.984519 8340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "f1492f21d20c4c748ea0e075b814eb9d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d0a22df32a21fae72c527ef23a0f6875"
server_key_iv: "b857b7e201c3693702219ff64ccebdb3"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.985023 8340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "f1492f21d20c4c748ea0e075b814eb9d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d0a22df32a21fae72c527ef23a0f6875"
server_key_iv: "b857b7e201c3693702219ff64ccebdb3"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.988647 8340 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:53:48.991185 8355 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:48.992538 8340 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:48.992671 8340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "f1492f21d20c4c748ea0e075b814eb9d"
format_stamp: "Formatted at 2025-09-05 07:53:48 on dist-test-slave-mnfb"
server_key: "d0a22df32a21fae72c527ef23a0f6875"
server_key_iv: "b857b7e201c3693702219ff64ccebdb3"
server_key_version: "encryptionkey@0"
I20250905 07:53:48.992779 8340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:49.003535 8340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:49.006796 8340 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:49.006995 8340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:49.007572 8340 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:49.008553 8340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:49.008605 8340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:49.008658 8340 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:49.008688 8340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:49.017747 8340 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:45091
I20250905 07:53:49.017769 8469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:45091 every 8 connection(s)
I20250905 07:53:49.018800 8340 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:49.022703 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8340
I20250905 07:53:49.022840 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:49.023074 4292 external_mini_cluster.cc:1468] Setting key fa8807d9000bd0cd067854d81025425f
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:49.031857 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:49.020702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:41809 (local address 127.4.49.62:36915)
0905 07:53:49.020862 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:49.020867 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:49.021690 (+ 823us) server_negotiation.cc:408] Connection header received
0905 07:53:49.022564 (+ 874us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:49.022568 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:49.022633 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:49.022733 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:49.024701 (+ 1968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.025347 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:49.026040 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.026197 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:49.028442 (+ 2245us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:49.028466 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:49.028468 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:49.028500 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:49.029906 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:49.030382 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:49.030384 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:49.030385 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:49.030450 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:49.030766 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:49.030771 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:49.030772 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:49.030967 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:49.031065 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:49.031591 (+ 526us) server_negotiation.cc:300] Negotiation successful
0905 07:53:49.031724 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250905 07:53:49.032570 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:49.021029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36915 (local address 127.4.49.3:41809)
0905 07:53:49.021533 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:49.021574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:49.022326 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:49.022919 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:49.022928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:49.023382 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:49.024554 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:49.024573 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.025469 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:49.025473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:49.025885 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:49.025893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.026060 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:49.026662 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:49.026687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:49.028243 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:49.030033 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:49.030039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:49.030051 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:49.030302 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:49.030514 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:49.030516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:49.030518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:49.030607 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:49.031073 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:49.031079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:49.031340 (+ 261us) client_negotiation.cc:769] Sending connection context
0905 07:53:49.031527 (+ 187us) client_negotiation.cc:240] Negotiation successful
0905 07:53:49.031780 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20250905 07:53:49.033847 8470 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36915
I20250905 07:53:49.034112 8470 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:49.034698 8470 heartbeater.cc:507] Master 127.4.49.62:36915 requested a full tablet report, sending...
I20250905 07:53:49.035871 7999 ts_manager.cc:194] Registered new tserver with Master: f1492f21d20c4c748ea0e075b814eb9d (127.4.49.3:45091)
I20250905 07:53:49.036603 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:41809
I20250905 07:53:49.037072 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058829, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058829, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:49.061932 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:49.053813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.4.49.62:36915)
0905 07:53:49.053965 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:49.053970 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:49.054082 (+ 112us) server_negotiation.cc:408] Connection header received
0905 07:53:49.054224 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:49.054226 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:49.054275 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:49.054344 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:49.055235 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.055745 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:49.056496 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:49.056688 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:49.058944 (+ 2256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:49.058970 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:49.058973 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:49.059007 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:49.060303 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:49.060788 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:49.060792 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:49.060793 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:49.060856 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:49.061108 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:49.061110 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:49.061111 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:49.061310 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:49.061433 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:49.061684 (+ 251us) server_negotiation.cc:300] Negotiation successful
0905 07:53:49.061798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20250905 07:53:49.062408 7998 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35122
I20250905 07:53:49.063336 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8065
I20250905 07:53:49.068581 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8203
I20250905 07:53:49.074007 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8340
I20250905 07:53:49.079146 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 7967
2025-09-05T07:53:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3024 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[8483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[8483](info): set up 2 sockets
Sep 05 07:53:49 dist-test-slave-mnfb krb5kdc[8483](info): commencing operation
krb5kdc: starting...
W20250905 07:53:50.923131 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:50 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:51.051997 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:46611
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:46611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:51.160713 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:51.160960 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:51.161017 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:51.164355 8499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:51.164438 8499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:51.164461 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:51.164479 8499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:51.164496 8499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:51.168972 8499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:46611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:51.170065 8499 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:51.170923 8499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:51.176640 8508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.176645 8505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:51.176754 8499 server_base.cc:1047] running on GCE node
W20250905 07:53:51.176640 8506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:51.177697 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:51.178720 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:51.179872 8499 hybrid_clock.cc:648] HybridClock initialized: now 1757058831179852 us; error 39 us; skew 500 ppm
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:51.182657 8499 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:51.183838 8499 webserver.cc:480] Webserver started at http://127.4.49.62:41983/ using document root <none> and password file <none>
I20250905 07:53:51.184464 8499 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:51.184521 8499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:51.184738 8499 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:51.186604 8499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "e74673e7404043f992778030d8dabae8"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "3ed144f0cfa1ea0db1a0d43d967f41fc"
server_key_iv: "1640f1b36f35b51a2187e724dd2709cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.187129 8499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "e74673e7404043f992778030d8dabae8"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "3ed144f0cfa1ea0db1a0d43d967f41fc"
server_key_iv: "1640f1b36f35b51a2187e724dd2709cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.190925 8499 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:51.193558 8515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.194681 8499 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:51.194808 8499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "e74673e7404043f992778030d8dabae8"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "3ed144f0cfa1ea0db1a0d43d967f41fc"
server_key_iv: "1640f1b36f35b51a2187e724dd2709cf"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.194922 8499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:51.210916 8499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:51.219164 8499 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:51.219374 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:51.226871 8499 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:46611
I20250905 07:53:51.226872 8568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:46611 every 8 connection(s)
I20250905 07:53:51.227979 8499 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:51.230974 8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:51.236901 8569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: Bootstrap starting.
I20250905 07:53:51.237911 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8499
I20250905 07:53:51.238037 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:51.238301 4292 external_mini_cluster.cc:1468] Setting key 14fb6edae58bc0279b8afe17bc556bd6
I20250905 07:53:51.239903 8569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:51.240732 8569 log.cc:826] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:51.243001 8569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: No bootstrap required, opened a new log
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:51.246286 8569 raft_consensus.cc:357] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } }
I20250905 07:53:51.246513 8569 raft_consensus.cc:383] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:51.246580 8569 raft_consensus.cc:738] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74673e7404043f992778030d8dabae8, State: Initialized, Role: FOLLOWER
I20250905 07:53:51.247108 8569 consensus_queue.cc:260] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } }
I20250905 07:53:51.247251 8569 raft_consensus.cc:397] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:51.247313 8569 raft_consensus.cc:491] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:51.247437 8569 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:51.248389 8569 raft_consensus.cc:513] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } }
I20250905 07:53:51.248752 8569 leader_election.cc:304] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e74673e7404043f992778030d8dabae8; no voters:
I20250905 07:53:51.249055 8569 leader_election.cc:290] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:51.249189 8574 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:51.249436 8574 raft_consensus.cc:695] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [term 1 LEADER]: Becoming Leader. State: Replica: e74673e7404043f992778030d8dabae8, State: Running, Role: LEADER
I20250905 07:53:51.249825 8574 consensus_queue.cc:237] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } }
I20250905 07:53:51.250286 8569 sys_catalog.cc:564] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:51.251117 8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e74673e7404043f992778030d8dabae8. Latest consensus state: current_term: 1 leader_uuid: "e74673e7404043f992778030d8dabae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } } }
I20250905 07:53:51.251240 8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:51.252554 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.239498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.4.49.62:46611)
0905 07:53:51.239975 (+ 477us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:51.239988 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:51.240036 (+ 48us) server_negotiation.cc:408] Connection header received
0905 07:53:51.240714 (+ 678us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:51.240734 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:51.241089 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:51.241498 (+ 409us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:51.242542 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.243403 (+ 861us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.244301 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.244567 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.247056 (+ 2489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:51.247082 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:51.247095 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:51.247139 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:51.249134 (+ 1995us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.249636 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.249642 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.249648 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.249756 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.250063 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.250066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.250068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.250554 (+ 486us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:51.251257 (+ 703us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:51.251490 (+ 233us) server_negotiation.cc:300] Negotiation successful
0905 07:53:51.251735 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":149,"threads_started":1}
I20250905 07:53:51.252823 8576 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e74673e7404043f992778030d8dabae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74673e7404043f992778030d8dabae8" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 46611 } } }
I20250905 07:53:51.252920 8576 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8 [sys.catalog]: This master's current role is: LEADER
W20250905 07:53:51.255102 8590 catalog_manager.cc:1560] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:51.255239 8590 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:51.255333 8584 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:51.256314 8584 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:51.262130 8584 catalog_manager.cc:1349] Generated new cluster ID: d972be73fe8347629fd206e974eb69c1
I20250905 07:53:51.262229 8584 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:51.275635 8584 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:51.277020 8584 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:51.284570 8584 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e74673e7404043f992778030d8dabae8: Generated new TSK 0
I20250905 07:53:51.285378 8584 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:51.333199 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:46611
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:51.447818 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:51.448153 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:51.448223 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:51.451750 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:51.451835 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:51.451920 8598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:51.456622 8598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:51.457792 8598 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:51.458735 8598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:51.465732 8606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.465711 8604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.465718 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:51.465935 8598 server_base.cc:1047] running on GCE node
I20250905 07:53:51.466425 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:51.467046 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:51.468206 8598 hybrid_clock.cc:648] HybridClock initialized: now 1757058831468188 us; error 33 us; skew 500 ppm
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:51.471163 8598 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:51.472398 8598 webserver.cc:480] Webserver started at http://127.4.49.1:39051/ using document root <none> and password file <none>
I20250905 07:53:51.472985 8598 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:51.473038 8598 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:51.473240 8598 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:51.475099 8598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "d3b93a28bbd44cc791a71fa27cb3b74f"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "c2f63be1c3d2e214ec2fce555b8be570"
server_key_iv: "b1f3a194f6744ab672f4dee96e8c383b"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.475620 8598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "d3b93a28bbd44cc791a71fa27cb3b74f"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "c2f63be1c3d2e214ec2fce555b8be570"
server_key_iv: "b1f3a194f6744ab672f4dee96e8c383b"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.479393 8598 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:51.481861 8613 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.482991 8598 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:53:51.483115 8598 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "d3b93a28bbd44cc791a71fa27cb3b74f"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "c2f63be1c3d2e214ec2fce555b8be570"
server_key_iv: "b1f3a194f6744ab672f4dee96e8c383b"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.483220 8598 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:51.495445 8598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:51.498469 8598 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:51.498656 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:51.499225 8598 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:51.500213 8598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:51.500263 8598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.500308 8598 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:51.500335 8598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.509021 8598 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:45183
I20250905 07:53:51.509045 8728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:45183 every 8 connection(s)
I20250905 07:53:51.510028 8598 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:51.519184 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8598
I20250905 07:53:51.519306 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:51.519586 4292 external_mini_cluster.cc:1468] Setting key e8dc11cbe9f8c83ec605e47f71a1cf5a
I20250905 07:53:51.523276 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.511848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:45711 (local address 127.4.49.62:46611)
0905 07:53:51.512014 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:51.512018 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:51.512778 (+ 760us) server_negotiation.cc:408] Connection header received
0905 07:53:51.513581 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:51.513584 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:51.513637 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:51.513718 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:51.515118 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.515646 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.516320 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.516469 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.519294 (+ 2825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:51.519325 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:51.519328 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:51.519365 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:51.520943 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.521598 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.521601 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.521603 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.521657 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.521987 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.521988 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.521990 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.522160 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:51.522253 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:51.522977 (+ 724us) server_negotiation.cc:300] Negotiation successful
0905 07:53:51.523130 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:53:51.524488 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.512171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:46611 (local address 127.4.49.1:45711)
0905 07:53:51.512614 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:51.512664 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:51.513392 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:51.513912 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:51.513922 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:51.514348 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:51.514960 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.514975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.515783 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.515787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:51.516223 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.516232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.516418 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.517498 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:51.517526 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:51.519068 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:51.521075 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.521083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.521099 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.521482 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.521749 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.521752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.521754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.521899 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.522246 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:51.522253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:51.522623 (+ 370us) client_negotiation.cc:769] Sending connection context
0905 07:53:51.523461 (+ 838us) client_negotiation.cc:240] Negotiation successful
0905 07:53:51.523758 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20250905 07:53:51.525913 8729 heartbeater.cc:344] Connected to a master server at 127.4.49.62:46611
I20250905 07:53:51.526178 8729 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:51.526786 8729 heartbeater.cc:507] Master 127.4.49.62:46611 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:53:51.528541 8532 ts_manager.cc:194] Registered new tserver with Master: d3b93a28bbd44cc791a71fa27cb3b74f (127.4.49.1:45183)
I20250905 07:53:51.529892 8532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:45711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:51.567667 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:46611
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:51.675650 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:51.675906 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:51.675967 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:51.679490 8736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:51.679577 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:51.679663 8736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:51.684326 8736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:51.685423 8736 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:51.686283 8736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:51.693866 8741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.693868 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.693905 8744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:51.694334 8736 server_base.cc:1047] running on GCE node
I20250905 07:53:51.694774 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:51.695356 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:51.696509 8736 hybrid_clock.cc:648] HybridClock initialized: now 1757058831696491 us; error 32 us; skew 500 ppm
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:51.699318 8736 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:51.700537 8736 webserver.cc:480] Webserver started at http://127.4.49.2:44933/ using document root <none> and password file <none>
I20250905 07:53:51.701089 8736 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:51.701136 8736 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:51.701305 8736 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:51.703120 8736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "1434a854d7004e67a35c84f0f7236bbd"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "0cdc083449be34dac86f264725a1875d"
server_key_iv: "d271d9080267b0c076e4778a4dc323ef"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.703574 8736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "1434a854d7004e67a35c84f0f7236bbd"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "0cdc083449be34dac86f264725a1875d"
server_key_iv: "d271d9080267b0c076e4778a4dc323ef"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.707196 8736 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:51.709606 8751 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.710738 8736 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:51.710851 8736 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "1434a854d7004e67a35c84f0f7236bbd"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "0cdc083449be34dac86f264725a1875d"
server_key_iv: "d271d9080267b0c076e4778a4dc323ef"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.710945 8736 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:51.733891 8736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:51.737170 8736 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:51.737406 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:51.737990 8736 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:51.738904 8736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:51.738956 8736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.739009 8736 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:51.739038 8736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.747773 8736 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:43661
I20250905 07:53:51.747803 8866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:43661 every 8 connection(s)
I20250905 07:53:51.748835 8736 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:51.753768 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8736
I20250905 07:53:51.753897 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:51.754154 4292 external_mini_cluster.cc:1468] Setting key 26f6221e63941ef0e2450c6d0f8bad77
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:51.763242 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.750855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:42557 (local address 127.4.49.62:46611)
0905 07:53:51.751007 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:51.751011 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:51.751873 (+ 862us) server_negotiation.cc:408] Connection header received
0905 07:53:51.752854 (+ 981us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:51.752858 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:51.752918 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:51.752992 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:51.754682 (+ 1690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.755180 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.755942 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.756143 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.759376 (+ 3233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:51.759400 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:51.759403 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:51.759451 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:51.761131 (+ 1680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.761681 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.761684 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.761687 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.761752 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.762009 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.762012 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.762014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.762239 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:51.762345 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:51.762916 (+ 571us) server_negotiation.cc:300] Negotiation successful
0905 07:53:51.763063 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:53:51.763859 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.751110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:46611 (local address 127.4.49.2:42557)
0905 07:53:51.751665 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:51.751764 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:51.752637 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:51.753176 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:51.753185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:51.753699 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:51.754510 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.754532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.755307 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.755310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:51.755821 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.755831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.756093 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.757321 (+ 1228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:51.757389 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:51.759083 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:51.761276 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.761282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.761293 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.761564 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.761837 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.761839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.761841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.761930 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.762334 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:51.762339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:51.762610 (+ 271us) client_negotiation.cc:769] Sending connection context
0905 07:53:51.762822 (+ 212us) client_negotiation.cc:240] Negotiation successful
0905 07:53:51.763075 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20250905 07:53:51.765069 8867 heartbeater.cc:344] Connected to a master server at 127.4.49.62:46611
I20250905 07:53:51.765312 8867 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:51.765866 8867 heartbeater.cc:507] Master 127.4.49.62:46611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:53:51.767042 8532 ts_manager.cc:194] Registered new tserver with Master: 1434a854d7004e67a35c84f0f7236bbd (127.4.49.2:43661)
I20250905 07:53:51.767769 8532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:42557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:51.799145 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:46611
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:51.906517 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:51.906786 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:51.906873 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:51.910344 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:51.910431 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:51.910542 8874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:51.915094 8874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:51.916265 8874 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:51.917161 8874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:51.923977 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.924036 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:51.923977 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:51.924206 8874 server_base.cc:1047] running on GCE node
I20250905 07:53:51.924685 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:51.925232 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:51.926401 8874 hybrid_clock.cc:648] HybridClock initialized: now 1757058831926378 us; error 35 us; skew 500 ppm
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:51.929399 8874 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:51.930516 8874 webserver.cc:480] Webserver started at http://127.4.49.3:37887/ using document root <none> and password file <none>
I20250905 07:53:51.931083 8874 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:51.931134 8874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:51.931318 8874 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:51.933159 8874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "0156061792b94709988462255137b6db"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "eb8e44f7587b9e3c0b9ffdf519987269"
server_key_iv: "33bea4b669599353dfa1bdea81e0b457"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.933641 8874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "0156061792b94709988462255137b6db"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "eb8e44f7587b9e3c0b9ffdf519987269"
server_key_iv: "33bea4b669599353dfa1bdea81e0b457"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.937161 8874 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250905 07:53:51.939505 8889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.940591 8874 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:51.940722 8874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "0156061792b94709988462255137b6db"
format_stamp: "Formatted at 2025-09-05 07:53:51 on dist-test-slave-mnfb"
server_key: "eb8e44f7587b9e3c0b9ffdf519987269"
server_key_iv: "33bea4b669599353dfa1bdea81e0b457"
server_key_version: "encryptionkey@0"
I20250905 07:53:51.940831 8874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:51.951881 8874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:51.954870 8874 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:51.955035 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:51.955564 8874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:51.956457 8874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:51.956506 8874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.956545 8874 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:51.956565 8874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:51.965351 8874 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:43057
I20250905 07:53:51.965375 9004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:43057 every 8 connection(s)
I20250905 07:53:51.966343 8874 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:51.975023 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 8874
I20250905 07:53:51.975129 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:51.975346 4292 external_mini_cluster.cc:1468] Setting key c1a46edd7251b41621b5d7df33b25843
I20250905 07:53:51.978806 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.968176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35371 (local address 127.4.49.62:46611)
0905 07:53:51.968311 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:51.968315 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:51.969054 (+ 739us) server_negotiation.cc:408] Connection header received
0905 07:53:51.969934 (+ 880us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:51.969938 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:51.970000 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:51.970074 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:51.971398 (+ 1324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.971905 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.972573 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.972754 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.975413 (+ 2659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:51.975441 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:51.975445 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:51.975478 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:51.977019 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.977602 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.977605 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.977607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.977656 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:51.977885 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:51.977887 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:51.977888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:51.978044 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:51.978113 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:51.978570 (+ 457us) server_negotiation.cc:300] Negotiation successful
0905 07:53:51.978672 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:53:51.979552 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:51.968457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:46611 (local address 127.4.49.3:35371)
0905 07:53:51.968908 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:51.968948 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:51.969711 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:51.970256 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:51.970265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:51.970676 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:51.971242 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.971256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.972021 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:51.972024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:51.972459 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:51.972467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:51.972670 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:51.973736 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:51.973762 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:51.975255 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:51.977129 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.977135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.977148 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.977500 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.977734 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:51.977737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:51.977738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:51.977825 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:51.978111 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:51.978116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:51.978365 (+ 249us) client_negotiation.cc:769] Sending connection context
0905 07:53:51.978546 (+ 181us) client_negotiation.cc:240] Negotiation successful
0905 07:53:51.978786 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20250905 07:53:51.980671 9005 heartbeater.cc:344] Connected to a master server at 127.4.49.62:46611
I20250905 07:53:51.980927 9005 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:51.981499 9005 heartbeater.cc:507] Master 127.4.49.62:46611 requested a full tablet report, sending...
I20250905 07:53:51.982560 8532 ts_manager.cc:194] Registered new tserver with Master: 0156061792b94709988462255137b6db (127.4.49.3:43057)
I20250905 07:53:51.983122 8532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:35371
I20250905 07:53:51.989355 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 05 07:53:51 dist-test-slave-mnfb krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 05 07:53:52 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:53:52.013231 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:52.004147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.4.49.1:45183)
0905 07:53:52.004435 (+ 288us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:52.004441 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:52.004468 (+ 27us) server_negotiation.cc:408] Connection header received
0905 07:53:52.004522 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:52.004527 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:52.004718 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:52.004847 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:52.005655 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:52.006391 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:52.007043 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:52.007238 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:52.009417 (+ 2179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:52.009468 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:52.009486 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:52.009519 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:52.011369 (+ 1850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:52.011842 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:52.011847 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:52.011849 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:52.011911 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:52.012191 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:52.012194 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:52.012195 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:52.012437 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:52.012611 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:52.012874 (+ 263us) server_negotiation.cc:300] Negotiation successful
0905 07:53:52.013034 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
Sep 05 07:53:52 dist-test-slave-mnfb krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:52.024019 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:52.016278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.4.49.62:46611)
0905 07:53:52.016416 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:52.016421 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:52.016437 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:53:52.016581 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:52.016584 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:52.016631 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:52.016709 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:52.017469 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:52.017989 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:52.018610 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:52.018808 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:52.021242 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:52.021267 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:52.021270 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:52.021304 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:52.022498 (+ 1194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:52.022904 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:52.022908 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:52.022910 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:52.022962 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:52.023172 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:52.023176 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:52.023178 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:52.023380 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:52.023503 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:52.023726 (+ 223us) server_negotiation.cc:300] Negotiation successful
0905 07:53:52.023856 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20250905 07:53:52.024422 8532 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52940
I20250905 07:53:52.025099 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8598
I20250905 07:53:52.031288 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8736
I20250905 07:53:52.036471 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8874
I20250905 07:53:52.041837 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 8499
2025-09-05T07:53:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2962 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:53:52 dist-test-slave-mnfb krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:53:52 dist-test-slave-mnfb krb5kdc[9017](info): set up 2 sockets
Sep 05 07:53:52 dist-test-slave-mnfb krb5kdc[9017](info): commencing operation
krb5kdc: starting...
W20250905 07:53:53.862820 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.799s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:53:53 dist-test-slave-mnfb krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:53:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:53.994786 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:42699
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:54.102072 9033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:54.102326 9033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:54.102404 9033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:54.105787 9033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:54.105865 9033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:54.105896 9033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:54.105921 9033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:54.105948 9033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:54.110431 9033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:54.111575 9033 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:54.112495 9033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:54.117779 9039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.117864 9041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.117936 9038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:54.118042 9033 server_base.cc:1047] running on GCE node
I20250905 07:53:54.118639 9033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:54.119576 9033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:54.120739 9033 hybrid_clock.cc:648] HybridClock initialized: now 1757058834120717 us; error 37 us; skew 500 ppm
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:54.123426 9033 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:53:54.124192 9033 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:54.124255 9033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:54.124460 9033 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:54.126191 9033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "1adc215815e847b19dafdc09fb433f75"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "79cbcc7f828bd66f46bc7ab14bf6b4cc"
server_key_iv: "ab88d472b116edf4b45669f0a0bd65c2"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.126658 9033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "1adc215815e847b19dafdc09fb433f75"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "79cbcc7f828bd66f46bc7ab14bf6b4cc"
server_key_iv: "ab88d472b116edf4b45669f0a0bd65c2"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.130064 9033 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:53:54.132390 9047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.133428 9033 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:54.133545 9033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "1adc215815e847b19dafdc09fb433f75"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "79cbcc7f828bd66f46bc7ab14bf6b4cc"
server_key_iv: "ab88d472b116edf4b45669f0a0bd65c2"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.133646 9033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:54.144841 9033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:54.147984 9033 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:54.148182 9033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:54.155068 9101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:42699 every 8 connection(s)
I20250905 07:53:54.155090 9033 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:42699
I20250905 07:53:54.155753 9033 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:54.158519 9102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:54.160645 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9033
I20250905 07:53:54.160753 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:54.160959 4292 external_mini_cluster.cc:1468] Setting key 53e1e655a8a1fc456c96509b61dc9ee6
I20250905 07:53:54.164276 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: Bootstrap starting.
I20250905 07:53:54.166381 9102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:54.167148 9102 log.cc:826] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: Log is configured to *not* fsync() on all Append() calls
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:54.169183 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: No bootstrap required, opened a new log
I20250905 07:53:54.172034 9102 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } }
I20250905 07:53:54.172333 9102 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:54.172977 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.162117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.4.49.62:42699)
0905 07:53:54.162537 (+ 420us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.162548 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.162583 (+ 35us) server_negotiation.cc:408] Connection header received
0905 07:53:54.163269 (+ 686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.163287 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.163590 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.163922 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:54.164811 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.165530 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.166154 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.166427 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.168648 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:54.168680 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:54.168692 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:54.168726 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:54.170400 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.170847 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.170851 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.170855 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.170920 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.171117 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.171119 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.171120 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.171429 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:54.171582 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.171911 (+ 329us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.172230 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20250905 07:53:54.172413 9102 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adc215815e847b19dafdc09fb433f75, State: Initialized, Role: FOLLOWER
I20250905 07:53:54.173722 9102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } }
I20250905 07:53:54.173874 9102 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:54.173959 9102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:54.174073 9102 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:54.174969 9102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } }
I20250905 07:53:54.175309 9102 leader_election.cc:304] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adc215815e847b19dafdc09fb433f75; no voters:
I20250905 07:53:54.175659 9102 leader_election.cc:290] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:54.175827 9107 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:54.176105 9107 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [term 1 LEADER]: Becoming Leader. State: Replica: 1adc215815e847b19dafdc09fb433f75, State: Running, Role: LEADER
I20250905 07:53:54.176445 9107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } }
I20250905 07:53:54.176934 9102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:54.178409 9109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1adc215815e847b19dafdc09fb433f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } } }
I20250905 07:53:54.178555 9109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:54.178850 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adc215815e847b19dafdc09fb433f75. Latest consensus state: current_term: 1 leader_uuid: "1adc215815e847b19dafdc09fb433f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc215815e847b19dafdc09fb433f75" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42699 } } }
I20250905 07:53:54.178907 9116 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:54.178964 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:54.182490 9116 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:54.189060 9116 catalog_manager.cc:1349] Generated new cluster ID: 9f3c80d9d17c441394328eee7ec4404a
I20250905 07:53:54.189162 9116 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:54.200156 9116 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:54.201514 9116 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:54.215544 9116 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: Generated new TSK 0
I20250905 07:53:54.216359 9116 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:54.265008 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42699
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:54.390200 9130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:54.390441 9130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:54.390515 9130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:54.393889 9130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:54.393976 9130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:54.394097 9130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:54.398609 9130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:54.399760 9130 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:54.400691 9130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:54.407051 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.407051 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.407051 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:54.407359 9130 server_base.cc:1047] running on GCE node
I20250905 07:53:54.407773 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:54.408385 9130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:54.409543 9130 hybrid_clock.cc:648] HybridClock initialized: now 1757058834409528 us; error 32 us; skew 500 ppm
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:54.412289 9130 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:53:54.413045 9130 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:54.413113 9130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:54.413309 9130 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:54.415127 9130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "8282e283bca24877b9fba8fc3f242c12"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "99672732c5474c05cb9c62256e6651c5"
server_key_iv: "b31c3eeeb58ab6dca0975be9fce2f572"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.415619 9130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "8282e283bca24877b9fba8fc3f242c12"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "99672732c5474c05cb9c62256e6651c5"
server_key_iv: "b31c3eeeb58ab6dca0975be9fce2f572"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.419121 9130 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:54.421554 9144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.422672 9130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250905 07:53:54.422798 9130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "8282e283bca24877b9fba8fc3f242c12"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "99672732c5474c05cb9c62256e6651c5"
server_key_iv: "b31c3eeeb58ab6dca0975be9fce2f572"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.422899 9130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:54.447679 9130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:54.450505 9130 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:54.450690 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:54.451233 9130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:54.452205 9130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:54.452255 9130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.452309 9130 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:54.452344 9130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.461397 9130 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:39323
I20250905 07:53:54.461407 9260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:39323 every 8 connection(s)
I20250905 07:53:54.461998 9130 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:54.471696 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9130
I20250905 07:53:54.471820 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:54.472085 4292 external_mini_cluster.cc:1468] Setting key b34d0d18ef6d662fe1b6480f444c7bef
I20250905 07:53:54.474313 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.463719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:36541 (local address 127.4.49.62:42699)
0905 07:53:54.463877 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.463881 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.464554 (+ 673us) server_negotiation.cc:408] Connection header received
0905 07:53:54.465413 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.465416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.465480 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.465554 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:54.467264 (+ 1710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.467783 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.468460 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.468635 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.470814 (+ 2179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:54.470838 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:54.470840 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:54.470869 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:54.472230 (+ 1361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.472785 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.472789 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.472791 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.472845 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.473109 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.473112 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.473115 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.473300 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:54.473399 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.474063 (+ 664us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.474179 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250905 07:53:54.475940 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.463962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42699 (local address 127.4.49.1:36541)
0905 07:53:54.464401 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.464448 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.465209 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.465739 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.465747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.466154 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:54.467102 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.467117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.467908 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.467911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.468342 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.468349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.468570 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.469187 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:54.469213 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:54.470650 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:54.472338 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.472344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.472358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.472689 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.472923 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.472926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.472928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.473029 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.473411 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:54.473419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:54.473769 (+ 350us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.474029 (+ 260us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.474356 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
I20250905 07:53:54.477166 9261 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42699
I20250905 07:53:54.477427 9261 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:54.478013 9261 heartbeater.cc:507] Master 127.4.49.62:42699 requested a full tablet report, sending...
I20250905 07:53:54.479679 9064 ts_manager.cc:194] Registered new tserver with Master: 8282e283bca24877b9fba8fc3f242c12 (127.4.49.1:39323)
I20250905 07:53:54.480901 9064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:36541
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:54.516458 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42699
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:54.625216 9268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:54.625483 9268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:54.625547 9268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:54.629069 9268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:54.629158 9268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:54.629251 9268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:54.633803 9268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:54.634949 9268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:54.635833 9268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:54.642343 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.642567 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.642341 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:54.642699 9268 server_base.cc:1047] running on GCE node
I20250905 07:53:54.643100 9268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:54.643685 9268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:54.644829 9268 hybrid_clock.cc:648] HybridClock initialized: now 1757058834644810 us; error 34 us; skew 500 ppm
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:54.647600 9268 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:53:54.648432 9268 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:54.648495 9268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:54.648669 9268 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:54.650473 9268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "05ba325abd20413f989ad1745c28dd00"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "65629aab733d6cdba8321804f00bef25"
server_key_iv: "b33c29db8a65c2a90ed7834edbc180d5"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.650918 9268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "05ba325abd20413f989ad1745c28dd00"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "65629aab733d6cdba8321804f00bef25"
server_key_iv: "b33c29db8a65c2a90ed7834edbc180d5"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.654302 9268 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250905 07:53:54.656594 9282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.657670 9268 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:53:54.657774 9268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "05ba325abd20413f989ad1745c28dd00"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "65629aab733d6cdba8321804f00bef25"
server_key_iv: "b33c29db8a65c2a90ed7834edbc180d5"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.657877 9268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:54.679751 9268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:54.682973 9268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:54.683144 9268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:54.683682 9268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:54.684607 9268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:54.684653 9268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.684705 9268 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:54.684731 9268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.693514 9268 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:34019
I20250905 07:53:54.693529 9398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:34019 every 8 connection(s)
I20250905 07:53:54.694103 9268 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:54.702215 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9268
I20250905 07:53:54.702312 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:54.702544 4292 external_mini_cluster.cc:1468] Setting key 4f48b081591746f18218322eda21c50f
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:54.706918 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.695732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:59869 (local address 127.4.49.62:42699)
0905 07:53:54.695879 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.695883 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.696577 (+ 694us) server_negotiation.cc:408] Connection header received
0905 07:53:54.697513 (+ 936us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.697517 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.697579 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.697674 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:54.699787 (+ 2113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.700513 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.701142 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.701272 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.703442 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:54.703454 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:54.703456 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:54.703483 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:54.705041 (+ 1558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.705525 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.705529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.705532 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.705588 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.705848 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.705852 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.705854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.706052 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:54.706151 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.706678 (+ 527us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.706790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:53:54.707811 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.695993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42699 (local address 127.4.49.2:59869)
0905 07:53:54.696442 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.696481 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.697247 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.697858 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.697870 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.698398 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:54.699610 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.699629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.700638 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.700641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.701050 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.701058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.701228 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.701775 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:54.701799 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:54.703303 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:54.705162 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.705167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.705179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.705423 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.705670 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.705673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.705675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.705762 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.706141 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:54.706146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:54.706399 (+ 253us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.706631 (+ 232us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.706946 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20250905 07:53:54.709007 9399 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42699
I20250905 07:53:54.709316 9399 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:54.709971 9399 heartbeater.cc:507] Master 127.4.49.62:42699 requested a full tablet report, sending...
I20250905 07:53:54.711145 9064 ts_manager.cc:194] Registered new tserver with Master: 05ba325abd20413f989ad1745c28dd00 (127.4.49.2:34019)
I20250905 07:53:54.711725 9064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:59869
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:53:54.746589 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42699
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:53:54.854063 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:54.854317 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:54.854410 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:54.857832 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:54.857919 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:54.858026 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:54.862556 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:54.863670 9406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:54.864567 9406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:54.870873 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:54.870874 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:54.871029 9406 server_base.cc:1047] running on GCE node
W20250905 07:53:54.870873 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:54.871557 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:54.872198 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:54.873368 9406 hybrid_clock.cc:648] HybridClock initialized: now 1757058834873349 us; error 32 us; skew 500 ppm
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:53:54.876139 9406 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:53:54.876886 9406 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:54.876945 9406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:54.877115 9406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:54.878944 9406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "3a99f3166a7d4736a43fea9934c65201"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "efcc9c569a0d395ca76e65d361898275"
server_key_iv: "6ed5e3ead4955f63d9e39ed176b97a2b"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.879400 9406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "3a99f3166a7d4736a43fea9934c65201"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "efcc9c569a0d395ca76e65d361898275"
server_key_iv: "6ed5e3ead4955f63d9e39ed176b97a2b"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.883024 9406 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:53:54.885392 9420 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.886670 9406 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:54.886785 9406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "3a99f3166a7d4736a43fea9934c65201"
format_stamp: "Formatted at 2025-09-05 07:53:54 on dist-test-slave-mnfb"
server_key: "efcc9c569a0d395ca76e65d361898275"
server_key_iv: "6ed5e3ead4955f63d9e39ed176b97a2b"
server_key_version: "encryptionkey@0"
I20250905 07:53:54.886894 9406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:54.909143 9406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:54.912346 9406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:54.912516 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:54.913097 9406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:54.913968 9406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:54.914017 9406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.914057 9406 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:54.914073 9406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:54.923405 9406 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:45921
I20250905 07:53:54.923456 9535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:45921 every 8 connection(s)
I20250905 07:53:54.924011 9406 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
Sep 05 07:53:54 dist-test-slave-mnfb krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:53:54.932510 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9406
I20250905 07:53:54.932610 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:54.932852 4292 external_mini_cluster.cc:1468] Setting key c5e6b67cb02713768d444ff94ba3a85f
I20250905 07:53:54.936321 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.925795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:44051 (local address 127.4.49.62:42699)
0905 07:53:54.925949 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.925953 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.926642 (+ 689us) server_negotiation.cc:408] Connection header received
0905 07:53:54.927481 (+ 839us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.927484 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.927537 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.927617 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:54.929406 (+ 1789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.929910 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.930547 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.930666 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.932864 (+ 2198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:54.932889 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:54.932893 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:54.932925 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:54.934444 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.934986 (+ 542us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.934989 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.934990 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.935038 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.935324 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.935327 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.935329 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.935492 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:54.935569 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.936025 (+ 456us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.936174 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250905 07:53:54.937059 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.926050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42699 (local address 127.4.49.3:44051)
0905 07:53:54.926491 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.926531 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.927283 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.927852 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.927861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.928283 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:54.929259 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.929275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.930020 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.930023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.930462 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.930469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.930630 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.931170 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:53:54.931192 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:54.932693 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:54.934560 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.934566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.934578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.934885 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.935119 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:53:54.935121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:53:54.935123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:53:54.935238 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:53:54.935573 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:54.935577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:54.935821 (+ 244us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.935998 (+ 177us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.936308 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20250905 07:53:54.938187 9536 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42699
I20250905 07:53:54.938443 9536 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:54.938958 9536 heartbeater.cc:507] Master 127.4.49.62:42699 requested a full tablet report, sending...
I20250905 07:53:54.939968 9064 ts_manager.cc:194] Registered new tserver with Master: 3a99f3166a7d4736a43fea9934c65201 (127.4.49.3:45921)
I20250905 07:53:54.940567 9064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:44051
I20250905 07:53:54.946700 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:53:54.955210 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.948582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.4.49.62:42699)
0905 07:53:54.948726 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.948730 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.948846 (+ 116us) server_negotiation.cc:408] Connection header received
0905 07:53:54.948976 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.948979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.949026 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.949090 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:54.950155 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.950643 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.951274 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.951456 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.952533 (+ 1077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:54.952552 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:54.952555 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:53:54.952583 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:54.953794 (+ 1211us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.954156 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.954159 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.954160 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.954208 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:53:54.954412 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:53:54.954416 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:53:54.954418 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:53:54.954604 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:54.954735 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.954936 (+ 201us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.955047 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:53:54.957940 9064 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:54.960357 9064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:54.973222 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.968762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:39323 (local address 127.0.0.1:49084)
0905 07:53:54.969109 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.969148 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.969277 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.970131 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.970137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.970155 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:54.970488 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.970497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.971971 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.971974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.972770 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.972780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.972876 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.972946 (+ 70us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.973026 (+ 80us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.973105 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20250905 07:53:54.974162 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.968881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.4.49.1:39323)
0905 07:53:54.969493 (+ 612us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.969500 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.969524 (+ 24us) server_negotiation.cc:408] Connection header received
0905 07:53:54.969612 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.969618 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.969807 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.969956 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:54.970814 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.971862 (+ 1048us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.973392 (+ 1530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.973846 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.973943 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.973999 (+ 56us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.974066 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":105,"threads_started":1}
I20250905 07:53:54.973641 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.969465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:45921 (local address 127.0.0.1:46458)
0905 07:53:54.970197 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.970210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.970287 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.970678 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.970682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.970698 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:54.970944 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.970952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.972399 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.972403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.973282 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.973291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.973396 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.973415 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.973461 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.973516 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":89,"threads_started":1}
I20250905 07:53:54.974576 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.968865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:34019 (local address 127.0.0.1:38664)
0905 07:53:54.970057 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:54.970071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:54.970149 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:54.970613 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:54.970616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:54.970641 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:54.970945 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.970952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.973389 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.973393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:54.974240 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:54.974250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.974350 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.974364 (+ 14us) client_negotiation.cc:769] Sending connection context
0905 07:53:54.974409 (+ 45us) client_negotiation.cc:240] Negotiation successful
0905 07:53:54.974463 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":105,"threads_started":1}
I20250905 07:53:54.974644 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.969846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.4.49.3:45921)
0905 07:53:54.970258 (+ 412us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.970264 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.970285 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:53:54.970417 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.970421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.970551 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.970685 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:54.971056 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.972106 (+ 1050us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.973744 (+ 1638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.974254 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.974362 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.974436 (+ 74us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.974525 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":125,"threads_started":1}
I20250905 07:53:54.975728 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:54.969512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.4.49.2:34019)
0905 07:53:54.969926 (+ 414us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:54.969932 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:54.970235 (+ 303us) server_negotiation.cc:408] Connection header received
0905 07:53:54.970321 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:54.970326 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:54.970480 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:54.970610 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:54.972156 (+ 1546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.973258 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:54.974833 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:54.975343 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:54.975455 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:54.975515 (+ 60us) server_negotiation.cc:300] Negotiation successful
0905 07:53:54.975599 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
I20250905 07:53:54.977272 9469 tablet_service.cc:1468] Processing CreateTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe (DEFAULT_TABLE table=test-table [id=b8101bc430914621936b487d8544b9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:54.977285 9192 tablet_service.cc:1468] Processing CreateTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe (DEFAULT_TABLE table=test-table [id=b8101bc430914621936b487d8544b9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:54.978178 9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a7171c4f504bc9822b98eb1f2bbcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:54.978224 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a7171c4f504bc9822b98eb1f2bbcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:54.978406 9331 tablet_service.cc:1468] Processing CreateTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe (DEFAULT_TABLE table=test-table [id=b8101bc430914621936b487d8544b9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:54.979257 9331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a7171c4f504bc9822b98eb1f2bbcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:54.983336 9553 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Bootstrap starting.
I20250905 07:53:54.985081 9555 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Bootstrap starting.
I20250905 07:53:54.985746 9553 tablet_bootstrap.cc:654] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:54.986735 9553 log.cc:826] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:54.987303 9555 tablet_bootstrap.cc:654] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:54.987483 9554 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: Bootstrap starting.
I20250905 07:53:54.988148 9555 log.cc:826] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:54.988590 9553 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: No bootstrap required, opened a new log
I20250905 07:53:54.988807 9553 ts_tablet_manager.cc:1397] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250905 07:53:54.989687 9554 tablet_bootstrap.cc:654] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:54.990118 9555 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: No bootstrap required, opened a new log
I20250905 07:53:54.990324 9555 ts_tablet_manager.cc:1397] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:53:54.990494 9554 log.cc:826] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:54.991854 9553 raft_consensus.cc:357] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.992164 9553 raft_consensus.cc:383] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:54.992260 9553 raft_consensus.cc:738] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a99f3166a7d4736a43fea9934c65201, State: Initialized, Role: FOLLOWER
I20250905 07:53:54.992699 9554 tablet_bootstrap.cc:492] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: No bootstrap required, opened a new log
I20250905 07:53:54.992744 9553 consensus_queue.cc:260] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.992902 9554 ts_tablet_manager.cc:1397] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250905 07:53:54.993467 9536 heartbeater.cc:499] Master 127.4.49.62:42699 was elected leader, sending a full tablet report...
I20250905 07:53:54.993782 9553 ts_tablet_manager.cc:1428] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250905 07:53:54.993743 9555 raft_consensus.cc:357] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.993948 9555 raft_consensus.cc:383] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:54.993993 9555 raft_consensus.cc:738] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ba325abd20413f989ad1745c28dd00, State: Initialized, Role: FOLLOWER
I20250905 07:53:54.994439 9555 consensus_queue.cc:260] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.995301 9555 ts_tablet_manager.cc:1428] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:53:54.995600 9554 raft_consensus.cc:357] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.995765 9554 raft_consensus.cc:383] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:54.995797 9554 raft_consensus.cc:738] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8282e283bca24877b9fba8fc3f242c12, State: Initialized, Role: FOLLOWER
I20250905 07:53:54.996253 9399 heartbeater.cc:499] Master 127.4.49.62:42699 was elected leader, sending a full tablet report...
I20250905 07:53:54.996368 9554 consensus_queue.cc:260] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:54.997105 9261 heartbeater.cc:499] Master 127.4.49.62:42699 was elected leader, sending a full tablet report...
I20250905 07:53:54.997395 9554 ts_tablet_manager.cc:1428] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 07:53:55.115204 9559 raft_consensus.cc:491] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:55.115453 9559 raft_consensus.cc:513] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:55.116659 9559 leader_election.cc:290] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8282e283bca24877b9fba8fc3f242c12 (127.4.49.1:39323), 05ba325abd20413f989ad1745c28dd00 (127.4.49.2:34019)
I20250905 07:53:55.119697 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.116961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:34019 (local address 127.4.49.3:35399)
0905 07:53:55.117089 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:55.117104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:55.117207 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:55.117558 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:55.117562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:55.117589 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:55.117878 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:55.117886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.118708 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:55.118712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:55.119346 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:55.119363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.119457 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:55.119476 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:53:55.119511 (+ 35us) client_negotiation.cc:240] Negotiation successful
0905 07:53:55.119565 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20250905 07:53:55.119964 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.116961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:39323 (local address 127.4.49.3:53963)
0905 07:53:55.117471 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:55.117484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:55.117567 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:55.117833 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:55.117837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:55.117855 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:53:55.118154 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:55.118162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.118924 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:55.118927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:55.119555 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:55.119563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.119680 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:55.119698 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:53:55.119744 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:53:55.119803 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":51,"thread_start_us":139,"threads_started":1}
I20250905 07:53:55.120340 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.117068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35399 (local address 127.4.49.2:34019)
0905 07:53:55.117261 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.117266 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.117281 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:53:55.117362 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.117366 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.117440 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.117524 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:55.118001 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.118595 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:55.119496 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.120010 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:55.120097 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.120154 (+ 57us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.120221 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250905 07:53:55.120411 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.117225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:53963 (local address 127.4.49.1:39323)
0905 07:53:55.117382 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.117386 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.117558 (+ 172us) server_negotiation.cc:408] Connection header received
0905 07:53:55.117658 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.117662 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.117716 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.117809 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:53:55.118263 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.118800 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:55.119686 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.120168 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:55.120208 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.120258 (+ 50us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.120317 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250905 07:53:55.120954 9351 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87a7171c4f504bc9822b98eb1f2bbcfe" candidate_uuid: "3a99f3166a7d4736a43fea9934c65201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ba325abd20413f989ad1745c28dd00" is_pre_election: true
I20250905 07:53:55.120978 9213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87a7171c4f504bc9822b98eb1f2bbcfe" candidate_uuid: "3a99f3166a7d4736a43fea9934c65201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8282e283bca24877b9fba8fc3f242c12" is_pre_election: true
I20250905 07:53:55.121232 9351 raft_consensus.cc:2466] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a99f3166a7d4736a43fea9934c65201 in term 0.
I20250905 07:53:55.121235 9213 raft_consensus.cc:2466] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a99f3166a7d4736a43fea9934c65201 in term 0.
I20250905 07:53:55.121695 9423 leader_election.cc:304] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05ba325abd20413f989ad1745c28dd00, 3a99f3166a7d4736a43fea9934c65201, 8282e283bca24877b9fba8fc3f242c12; no voters:
I20250905 07:53:55.121956 9559 raft_consensus.cc:2802] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:55.122027 9559 raft_consensus.cc:491] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:55.122057 9559 raft_consensus.cc:3058] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:55.123064 9559 raft_consensus.cc:513] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:55.123456 9559 leader_election.cc:290] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [CANDIDATE]: Term 1 election: Requested vote from peers 8282e283bca24877b9fba8fc3f242c12 (127.4.49.1:39323), 05ba325abd20413f989ad1745c28dd00 (127.4.49.2:34019)
I20250905 07:53:55.123853 9213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87a7171c4f504bc9822b98eb1f2bbcfe" candidate_uuid: "3a99f3166a7d4736a43fea9934c65201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8282e283bca24877b9fba8fc3f242c12"
I20250905 07:53:55.123860 9351 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87a7171c4f504bc9822b98eb1f2bbcfe" candidate_uuid: "3a99f3166a7d4736a43fea9934c65201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ba325abd20413f989ad1745c28dd00"
I20250905 07:53:55.123986 9213 raft_consensus.cc:3058] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:55.123986 9351 raft_consensus.cc:3058] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:55.125288 9351 raft_consensus.cc:2466] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a99f3166a7d4736a43fea9934c65201 in term 1.
I20250905 07:53:55.125692 9213 raft_consensus.cc:2466] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a99f3166a7d4736a43fea9934c65201 in term 1.
I20250905 07:53:55.125692 9424 leader_election.cc:304] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ba325abd20413f989ad1745c28dd00, 3a99f3166a7d4736a43fea9934c65201; no voters:
I20250905 07:53:55.125872 9559 raft_consensus.cc:2802] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:55.126121 9559 raft_consensus.cc:695] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 1 LEADER]: Becoming Leader. State: Replica: 3a99f3166a7d4736a43fea9934c65201, State: Running, Role: LEADER
I20250905 07:53:55.126434 9559 consensus_queue.cc:237] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } }
I20250905 07:53:55.130056 9064 catalog_manager.cc:5582] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a99f3166a7d4736a43fea9934c65201 (127.4.49.3). New cstate: current_term: 1 leader_uuid: "3a99f3166a7d4736a43fea9934c65201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a99f3166a7d4736a43fea9934c65201" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 45921 } health_report { overall_health: HEALTHY } } }
I20250905 07:53:55.171398 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.168385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.4.49.3:45921)
0905 07:53:55.168502 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.168506 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.168519 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:53:55.168633 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.168635 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.168706 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.168786 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:53:55.169089 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.169612 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:55.170278 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:55.170422 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:55.170516 (+ 94us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:53:55.171004 (+ 488us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:53:55.171102 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.171224 (+ 122us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.171284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20250905 07:53:55.174793 9537 tablet.cc:2378] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:55.177395 9213 raft_consensus.cc:1273] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 1 FOLLOWER]: Refusing update from remote peer 3a99f3166a7d4736a43fea9934c65201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:55.177376 9351 raft_consensus.cc:1273] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 1 FOLLOWER]: Refusing update from remote peer 3a99f3166a7d4736a43fea9934c65201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:55.178193 9559 consensus_queue.cc:1035] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8282e283bca24877b9fba8fc3f242c12" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:55.178427 9563 consensus_queue.cc:1035] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ba325abd20413f989ad1745c28dd00" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:55.185015 9566 mvcc.cc:204] Tried to move back new op lower bound from 7196912988878647296 to 7196912988680970240. Current Snapshot: MvccSnapshot[applied={T|T < 7196912988878647296}]
I20250905 07:53:55.185684 9568 mvcc.cc:204] Tried to move back new op lower bound from 7196912988878647296 to 7196912988680970240. Current Snapshot: MvccSnapshot[applied={T|T < 7196912988878647296}]
I20250905 07:53:55.193218 9062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:55.193420 9062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37070:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:55.195884 9062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1adc215815e847b19dafdc09fb433f75: Sending DeleteTablet for 3 replicas of tablet 87a7171c4f504bc9822b98eb1f2bbcfe
I20250905 07:53:55.196765 9192 tablet_service.cc:1515] Processing DeleteTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:55 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:49084
I20250905 07:53:55.196790 9469 tablet_service.cc:1515] Processing DeleteTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:55 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:46458
I20250905 07:53:55.196836 9331 tablet_service.cc:1515] Processing DeleteTablet for tablet 87a7171c4f504bc9822b98eb1f2bbcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:55 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:38664
I20250905 07:53:55.197288 9576 tablet_replica.cc:331] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12: stopping tablet replica
I20250905 07:53:55.197485 9578 tablet_replica.cc:331] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: stopping tablet replica
I20250905 07:53:55.197602 9576 raft_consensus.cc:2241] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:55.197805 9578 raft_consensus.cc:2241] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:55.197952 9576 raft_consensus.cc:2270] T 87a7171c4f504bc9822b98eb1f2bbcfe P 8282e283bca24877b9fba8fc3f242c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:55.197989 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9130
I20250905 07:53:55.198163 9578 raft_consensus.cc:2270] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:55.198599 9577 tablet_replica.cc:331] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: stopping tablet replica
I20250905 07:53:55.198777 9577 raft_consensus.cc:2241] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:53:55.199026 9577 raft_consensus.cc:2270] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:55.200346 9577 ts_tablet_manager.cc:1905] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:55.200768 9578 ts_tablet_manager.cc:1905] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:55.202998 9577 ts_tablet_manager.cc:1918] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:55.202998 9578 ts_tablet_manager.cc:1918] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:55.203080 9578 log.cc:1199] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/87a7171c4f504bc9822b98eb1f2bbcfe
I20250905 07:53:55.203080 9577 log.cc:1199] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/87a7171c4f504bc9822b98eb1f2bbcfe
I20250905 07:53:55.203348 9578 ts_tablet_manager.cc:1939] T 87a7171c4f504bc9822b98eb1f2bbcfe P 05ba325abd20413f989ad1745c28dd00: Deleting consensus metadata
I20250905 07:53:55.203348 9577 ts_tablet_manager.cc:1939] T 87a7171c4f504bc9822b98eb1f2bbcfe P 3a99f3166a7d4736a43fea9934c65201: Deleting consensus metadata
I20250905 07:53:55.204576 9050 catalog_manager.cc:4928] TS 3a99f3166a7d4736a43fea9934c65201 (127.4.49.3:45921): tablet 87a7171c4f504bc9822b98eb1f2bbcfe (table test-table [id=b8101bc430914621936b487d8544b9cf]) successfully deleted
I20250905 07:53:55.204576 9051 catalog_manager.cc:4928] TS 05ba325abd20413f989ad1745c28dd00 (127.4.49.2:34019): tablet 87a7171c4f504bc9822b98eb1f2bbcfe (table test-table [id=b8101bc430914621936b487d8544b9cf]) successfully deleted
W20250905 07:53:55.207657 9050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:39323 (error 108)
I20250905 07:53:55.208458 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9268
I20250905 07:53:55.209244 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.208561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:39323 (local address 127.0.0.1:49088)
0905 07:53:55.209079 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:55.209146 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:39323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
W20250905 07:53:55.209806 9050 catalog_manager.cc:4671] TS 8282e283bca24877b9fba8fc3f242c12 (127.4.49.1:39323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87a7171c4f504bc9822b98eb1f2bbcfe: Network error: Client connection negotiation failed: client connection to 127.4.49.1:39323: connect: Connection refused (error 111)
I20250905 07:53:55.215010 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9406
I20250905 07:53:55.220407 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9033
2025-09-05T07:53:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3178 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-09-05T07:53:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:55Z Disabled control of system clock
I20250905 07:53:55.247884 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39957
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:39957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250905 07:53:55.353734 9583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:55.353981 9583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:55.354028 9583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:55.357430 9583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:55.357512 9583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:55.357527 9583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:55.357546 9583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:55.357563 9583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:53:55.361619 9583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:39957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:55.362820 9583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:55.363993 9583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:55.369843 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.369943 9583 server_base.cc:1047] running on GCE node
W20250905 07:53:55.369843 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:55.369875 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.370607 9583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:55.371580 9583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:55.372749 9583 hybrid_clock.cc:648] HybridClock initialized: now 1757058835372736 us; error 36 us; skew 500 ppm
I20250905 07:53:55.374632 9583 webserver.cc:480] Webserver started at http://127.4.49.62:39343/ using document root <none> and password file <none>
I20250905 07:53:55.375219 9583 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:55.375276 9583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:55.375456 9583 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:55.377313 9583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "161179de6f4f4887a4c2e3e613973c48"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "fc02a88378ddf67827ecf9dc9744ab55"
server_key_iv: "02ae024075d4c292d230ca15af643517"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.377815 9583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "161179de6f4f4887a4c2e3e613973c48"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "fc02a88378ddf67827ecf9dc9744ab55"
server_key_iv: "02ae024075d4c292d230ca15af643517"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.381250 9583 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:55.383594 9597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.384750 9583 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:55.384871 9583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "161179de6f4f4887a4c2e3e613973c48"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "fc02a88378ddf67827ecf9dc9744ab55"
server_key_iv: "02ae024075d4c292d230ca15af643517"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.384989 9583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:55.389899 9583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:55.390516 9583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:55.390715 9583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:55.397675 9651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:39957 every 8 connection(s)
I20250905 07:53:55.397671 9583 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:39957
I20250905 07:53:55.398803 9583 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:55.401885 9652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:55.403062 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9583
I20250905 07:53:55.403192 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:55.403456 4292 external_mini_cluster.cc:1468] Setting key d62882a952f7dc520dc6d3f6bd6e817f
I20250905 07:53:55.408263 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: Bootstrap starting.
I20250905 07:53:55.408824 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.404887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.4.49.62:39957)
0905 07:53:55.405603 (+ 716us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.405613 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.405649 (+ 36us) server_negotiation.cc:408] Connection header received
0905 07:53:55.406180 (+ 531us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.406197 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.406530 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.406838 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:55.407665 (+ 827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:55.407670 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:55.407680 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:55.407691 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:55.407734 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:55.407745 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:55.407878 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.407965 (+ 87us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.408220 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":150,"threads_started":1}
I20250905 07:53:55.410626 9652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:55.411406 9652 log.cc:826] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:55.413240 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: No bootstrap required, opened a new log
I20250905 07:53:55.415869 9652 raft_consensus.cc:357] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } }
I20250905 07:53:55.416105 9652 raft_consensus.cc:383] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:55.416175 9652 raft_consensus.cc:738] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161179de6f4f4887a4c2e3e613973c48, State: Initialized, Role: FOLLOWER
I20250905 07:53:55.416585 9652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } }
I20250905 07:53:55.416710 9652 raft_consensus.cc:397] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:55.416770 9652 raft_consensus.cc:491] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:55.416874 9652 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:55.417886 9652 raft_consensus.cc:513] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } }
I20250905 07:53:55.418226 9652 leader_election.cc:304] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161179de6f4f4887a4c2e3e613973c48; no voters:
I20250905 07:53:55.418537 9652 leader_election.cc:290] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:55.418654 9657 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:55.418854 9657 raft_consensus.cc:695] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [term 1 LEADER]: Becoming Leader. State: Replica: 161179de6f4f4887a4c2e3e613973c48, State: Running, Role: LEADER
I20250905 07:53:55.419148 9657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } }
I20250905 07:53:55.419641 9652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:55.420789 9658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161179de6f4f4887a4c2e3e613973c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } } }
I20250905 07:53:55.420811 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161179de6f4f4887a4c2e3e613973c48. Latest consensus state: current_term: 1 leader_uuid: "161179de6f4f4887a4c2e3e613973c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161179de6f4f4887a4c2e3e613973c48" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39957 } } }
I20250905 07:53:55.421029 9658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:55.421056 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:55.421531 9667 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:55.425000 9667 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:55.431243 9667 catalog_manager.cc:1349] Generated new cluster ID: 048dbb2be6344623b7cf4e354764e0dc
I20250905 07:53:55.431336 9667 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:55.444546 9667 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:55.444888 9667 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:55.454558 9667 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: Generated new TSK 0
I20250905 07:53:55.455318 9667 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:53:55.467821 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--builtin_ntp_servers=127.4.49.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250905 07:53:55.576040 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:55.576340 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:55.576431 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:55.579849 9677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:55.579926 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:55.580029 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:53:55.584136 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:55.585296 9677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:55.586474 9677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:55.593274 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:55.593261 9685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.593393 9677 server_base.cc:1047] running on GCE node
W20250905 07:53:55.593262 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.593928 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:55.594518 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:55.595685 9677 hybrid_clock.cc:648] HybridClock initialized: now 1757058835595666 us; error 52 us; skew 500 ppm
I20250905 07:53:55.598157 9677 webserver.cc:480] Webserver started at http://127.4.49.1:42495/ using document root <none> and password file <none>
I20250905 07:53:55.598891 9677 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:55.598963 9677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:55.599170 9677 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:55.601455 9677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "36fb8aa2d41f407e8d35ccf0a3de500b"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "f161c7a7cc1bd8f6a7a334169924bc55"
server_key_iv: "c641bbfbbce9d62ed0ca43df8a694cf8"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.601959 9677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "36fb8aa2d41f407e8d35ccf0a3de500b"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "f161c7a7cc1bd8f6a7a334169924bc55"
server_key_iv: "c641bbfbbce9d62ed0ca43df8a694cf8"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.605429 9677 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:53:55.607733 9691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.608834 9677 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:55.608944 9677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "36fb8aa2d41f407e8d35ccf0a3de500b"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "f161c7a7cc1bd8f6a7a334169924bc55"
server_key_iv: "c641bbfbbce9d62ed0ca43df8a694cf8"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.609052 9677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:55.613850 9677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:55.614459 9677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:55.614614 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:55.615159 9677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:55.616142 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:55.616189 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.616233 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:55.616261 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.626639 9677 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:44745
I20250905 07:53:55.626684 9805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:44745 every 8 connection(s)
I20250905 07:53:55.627657 9677 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:53:55.633390 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9677
I20250905 07:53:55.633512 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:53:55.633790 4292 external_mini_cluster.cc:1468] Setting key db4bed8de631f2dc8d891e3cb30e967f
I20250905 07:53:55.633790 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.629755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:40205 (local address 127.4.49.62:39957)
0905 07:53:55.629919 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.629924 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.630991 (+ 1067us) server_negotiation.cc:408] Connection header received
0905 07:53:55.631819 (+ 828us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.631824 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.631878 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.631986 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:55.633186 (+ 1200us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:55.633191 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:55.633194 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:55.633213 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:55.633240 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:55.633252 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:55.633360 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.633569 (+ 209us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.633647 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250905 07:53:55.634961 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.630077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39957 (local address 127.4.49.1:40205)
0905 07:53:55.630827 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:55.630873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:55.631612 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:55.632166 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:55.632175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:55.632266 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:55.632903 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:55.632905 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:55.632939 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:55.632999 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:55.633348 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:55.633360 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:55.633366 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:53:55.634024 (+ 658us) client_negotiation.cc:240] Negotiation successful
0905 07:53:55.634257 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20250905 07:53:55.636082 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--builtin_ntp_servers=127.4.49.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20250905 07:53:55.636699 9807 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39957
I20250905 07:53:55.637001 9807 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:55.637473 9807 heartbeater.cc:507] Master 127.4.49.62:39957 requested a full tablet report, sending...
I20250905 07:53:55.639103 9615 ts_manager.cc:194] Registered new tserver with Master: 36fb8aa2d41f407e8d35ccf0a3de500b (127.4.49.1:44745)
W20250905 07:53:55.774660 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:55.774897 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:55.774952 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:55.778388 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:55.778465 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:55.778553 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:53:55.782617 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:55.783764 9810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:55.784977 9810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:55.792114 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:55.792114 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.792471 9810 server_base.cc:1047] running on GCE node
W20250905 07:53:55.792691 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.793072 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:55.793677 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:55.794857 9810 hybrid_clock.cc:648] HybridClock initialized: now 1757058835794834 us; error 36 us; skew 500 ppm
I20250905 07:53:55.796905 9810 webserver.cc:480] Webserver started at http://127.4.49.2:34987/ using document root <none> and password file <none>
I20250905 07:53:55.797523 9810 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:55.797581 9810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:55.797814 9810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:55.799700 9810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "35952083cc934170af7b23d9a4d87595"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "ec9714442c88783a1ced63d9862b881f"
server_key_iv: "8126dfc7ff7441e923d6e901e3ae6af1"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.800204 9810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "35952083cc934170af7b23d9a4d87595"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "ec9714442c88783a1ced63d9862b881f"
server_key_iv: "8126dfc7ff7441e923d6e901e3ae6af1"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.803738 9810 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:53:55.806207 9824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.807367 9810 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:55.807492 9810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "35952083cc934170af7b23d9a4d87595"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "ec9714442c88783a1ced63d9862b881f"
server_key_iv: "8126dfc7ff7441e923d6e901e3ae6af1"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.807608 9810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:55.812554 9810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:55.813198 9810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:55.813369 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:55.813925 9810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:55.814888 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:55.814937 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.814977 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:55.814991 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.823984 9810 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:37879
I20250905 07:53:55.824012 9938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:37879 every 8 connection(s)
I20250905 07:53:55.825017 9810 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:53:55.830489 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.826847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:55619 (local address 127.4.49.62:39957)
0905 07:53:55.826999 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.827004 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:55.827970 (+ 966us) server_negotiation.cc:408] Connection header received
0905 07:53:55.828830 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:55.828833 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:55.828876 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:55.828956 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:55.830020 (+ 1064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:55.830023 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:55.830025 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:55.830038 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:55.830060 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:55.830068 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:55.830152 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:55.830303 (+ 151us) server_negotiation.cc:300] Negotiation successful
0905 07:53:55.830346 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:53:55.831211 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.827157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39957 (local address 127.4.49.2:55619)
0905 07:53:55.827826 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:55.827865 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:55.828620 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:55.829100 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:55.829108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:55.829198 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:55.829780 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:55.829782 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:55.829817 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:55.829876 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:55.830151 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:55.830158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:55.830163 (+ 5us) client_negotiation.cc:769] Sending connection context
0905 07:53:55.830319 (+ 156us) client_negotiation.cc:240] Negotiation successful
0905 07:53:55.830522 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20250905 07:53:55.832373 9939 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39957
I20250905 07:53:55.832623 9939 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:55.833072 9939 heartbeater.cc:507] Master 127.4.49.62:39957 requested a full tablet report, sending...
I20250905 07:53:55.833339 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9810
I20250905 07:53:55.833439 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:53:55.833763 4292 external_mini_cluster.cc:1468] Setting key c6bd3e6e06a2521036c749f3ac01a235
I20250905 07:53:55.834244 9615 ts_manager.cc:194] Registered new tserver with Master: 35952083cc934170af7b23d9a4d87595 (127.4.49.2:37879)
I20250905 07:53:55.835646 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--builtin_ntp_servers=127.4.49.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250905 07:53:55.946775 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:55.947024 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:55.947077 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:55.950544 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:55.950618 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:55.950697 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:53:55.954668 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:55.955797 9942 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:55.957042 9942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:55.963606 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:55.963599 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:55.963599 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:55.963892 9942 server_base.cc:1047] running on GCE node
I20250905 07:53:55.964385 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:55.964959 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:55.966121 9942 hybrid_clock.cc:648] HybridClock initialized: now 1757058835966101 us; error 37 us; skew 500 ppm
I20250905 07:53:55.968158 9942 webserver.cc:480] Webserver started at http://127.4.49.3:36629/ using document root <none> and password file <none>
I20250905 07:53:55.968760 9942 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:55.968822 9942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:55.969013 9942 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:55.970800 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "3c9d9322981f4e8b95df9535665fc8b5"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "6fd53f8798be020eae867142d4178c0d"
server_key_iv: "09b684c6020491d12dc19f5eb6f8e19c"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.971334 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "3c9d9322981f4e8b95df9535665fc8b5"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "6fd53f8798be020eae867142d4178c0d"
server_key_iv: "09b684c6020491d12dc19f5eb6f8e19c"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.975404 9942 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 07:53:55.978104 9956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.979292 9942 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:53:55.979453 9942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "3c9d9322981f4e8b95df9535665fc8b5"
format_stamp: "Formatted at 2025-09-05 07:53:55 on dist-test-slave-mnfb"
server_key: "6fd53f8798be020eae867142d4178c0d"
server_key_iv: "09b684c6020491d12dc19f5eb6f8e19c"
server_key_version: "encryptionkey@0"
I20250905 07:53:55.979562 9942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:55.984753 9942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:55.985463 9942 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:55.985641 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:55.986188 9942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:53:55.987182 9942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:53:55.987231 9942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.987284 9942 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:53:55.987309 9942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:55.996451 9942 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:37047
I20250905 07:53:55.996532 10072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:37047 every 8 connection(s)
I20250905 07:53:55.997534 9942 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:53:56.001307 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 9942
I20250905 07:53:56.001430 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:53:56.001725 4292 external_mini_cluster.cc:1468] Setting key 45ff15adb294282484ac5b68fe3da627
I20250905 07:53:56.003273 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.999384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:41739 (local address 127.4.49.62:39957)
0905 07:53:55.999549 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:55.999554 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.000616 (+ 1062us) server_negotiation.cc:408] Connection header received
0905 07:53:56.001556 (+ 940us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.001560 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.001603 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.001670 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.002804 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.002807 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.002809 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.002822 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.002844 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.002852 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.002936 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.003098 (+ 162us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.003141 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250905 07:53:56.004037 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:55.999695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39957 (local address 127.4.49.3:41739)
0905 07:53:56.000466 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.000507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.001320 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.001848 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.001861 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.001976 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.002559 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.002561 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.002596 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.002657 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.002939 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.002949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.002955 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.003139 (+ 184us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.003369 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20250905 07:53:56.005226 10073 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39957
I20250905 07:53:56.005482 10073 heartbeater.cc:461] Registering TS with master...
I20250905 07:53:56.005904 10073 heartbeater.cc:507] Master 127.4.49.62:39957 requested a full tablet report, sending...
I20250905 07:53:56.006944 9614 ts_manager.cc:194] Registered new tserver with Master: 3c9d9322981f4e8b95df9535665fc8b5 (127.4.49.3:37047)
I20250905 07:53:56.015655 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:53:56.019078 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.017590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.4.49.62:39957)
0905 07:53:56.017715 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.017718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.017831 (+ 113us) server_negotiation.cc:408] Connection header received
0905 07:53:56.017945 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.017947 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.017983 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.018056 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.018708 (+ 652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.018711 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.018713 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.018724 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.018743 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.018750 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.018822 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.018923 (+ 101us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.018959 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:53:56.021972 9614 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:53:56.024396 9614 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:53:56.036470 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.033558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:44745 (local address 127.0.0.1:58238)
0905 07:53:56.034163 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.034178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.034262 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.034926 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.034930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.034945 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.035582 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.035585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.035630 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.035698 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.036074 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.036082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.036088 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.036214 (+ 126us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.036293 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":103,"threads_started":1}
I20250905 07:53:56.036476 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.033781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.4.49.1:44745)
0905 07:53:56.034503 (+ 722us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.034509 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.034533 (+ 24us) server_negotiation.cc:408] Connection header received
0905 07:53:56.034590 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.034596 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.034773 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.034918 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.035854 (+ 936us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.035861 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.035873 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.035900 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.035959 (+ 59us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.035973 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.036139 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.036226 (+ 87us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.036328 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":127,"threads_started":1}
I20250905 07:53:56.037253 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.032811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:37879 (local address 127.0.0.1:59030)
0905 07:53:56.033173 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.033219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.033312 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.034497 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.034501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.034555 (+ 54us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.035582 (+ 1027us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.035585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.035630 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.035698 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.036996 (+ 1298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.037000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.037002 (+ 2us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.037087 (+ 85us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.037125 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20250905 07:53:56.037984 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.034063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.4.49.3:37047)
0905 07:53:56.034754 (+ 691us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.034759 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.034784 (+ 25us) server_negotiation.cc:408] Connection header received
0905 07:53:56.034839 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.034844 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.034979 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.035094 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.036998 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.037006 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.037015 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.037028 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.037089 (+ 61us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.037102 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.037204 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.037757 (+ 553us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.037837 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":165,"threads_started":1}
I20250905 07:53:56.037989 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.032941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.4.49.2:37879)
0905 07:53:56.033317 (+ 376us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.033323 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.033378 (+ 55us) server_negotiation.cc:408] Connection header received
0905 07:53:56.033442 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.033448 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.033655 (+ 207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.033802 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.035873 (+ 2071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.035878 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.035885 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.035897 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.035944 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.035954 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.036956 (+ 1002us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.037547 (+ 591us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.037628 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":153,"threads_started":1}
I20250905 07:53:56.037495 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.033894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37047 (local address 127.0.0.1:58226)
0905 07:53:56.034285 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.034297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.034379 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.035819 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.035824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.035839 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.036829 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.036831 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.036844 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.036868 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.037246 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.037249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.037252 (+ 3us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.037317 (+ 65us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.037372 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":85,"threads_started":1}
I20250905 07:53:56.041471 10006 tablet_service.cc:1468] Processing CreateTablet for tablet 061650d7f2a548cbbafa23f7575cec50 (DEFAULT_TABLE table=test-table [id=c18c00f1624043ec954644594ecbd458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:56.041471 9873 tablet_service.cc:1468] Processing CreateTablet for tablet 061650d7f2a548cbbafa23f7575cec50 (DEFAULT_TABLE table=test-table [id=c18c00f1624043ec954644594ecbd458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:56.041508 9740 tablet_service.cc:1468] Processing CreateTablet for tablet 061650d7f2a548cbbafa23f7575cec50 (DEFAULT_TABLE table=test-table [id=c18c00f1624043ec954644594ecbd458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:53:56.042551 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061650d7f2a548cbbafa23f7575cec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:56.042553 10006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061650d7f2a548cbbafa23f7575cec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:56.042552 9873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061650d7f2a548cbbafa23f7575cec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:56.048951 10091 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Bootstrap starting.
I20250905 07:53:56.049813 10093 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Bootstrap starting.
I20250905 07:53:56.049813 10092 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Bootstrap starting.
I20250905 07:53:56.051968 10091 tablet_bootstrap.cc:654] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:56.052004 10092 tablet_bootstrap.cc:654] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:56.052004 10093 tablet_bootstrap.cc:654] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:56.052886 10091 log.cc:826] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:56.052896 10092 log.cc:826] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:56.052896 10093 log.cc:826] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:56.055043 10093 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: No bootstrap required, opened a new log
I20250905 07:53:56.055084 10091 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: No bootstrap required, opened a new log
I20250905 07:53:56.055101 10092 tablet_bootstrap.cc:492] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: No bootstrap required, opened a new log
I20250905 07:53:56.055280 10091 ts_tablet_manager.cc:1397] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250905 07:53:56.055279 10092 ts_tablet_manager.cc:1397] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250905 07:53:56.055279 10093 ts_tablet_manager.cc:1397] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250905 07:53:56.058446 10093 raft_consensus.cc:357] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.058445 10092 raft_consensus.cc:357] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.058708 10092 raft_consensus.cc:383] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:56.058708 10093 raft_consensus.cc:383] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:56.058760 10093 raft_consensus.cc:738] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9d9322981f4e8b95df9535665fc8b5, State: Initialized, Role: FOLLOWER
I20250905 07:53:56.058770 10092 raft_consensus.cc:738] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35952083cc934170af7b23d9a4d87595, State: Initialized, Role: FOLLOWER
I20250905 07:53:56.058679 10091 raft_consensus.cc:357] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.058914 10091 raft_consensus.cc:383] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:56.058979 10091 raft_consensus.cc:738] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fb8aa2d41f407e8d35ccf0a3de500b, State: Initialized, Role: FOLLOWER
I20250905 07:53:56.059284 10092 consensus_queue.cc:260] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.059284 10093 consensus_queue.cc:260] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.059497 10091 consensus_queue.cc:260] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.060242 9807 heartbeater.cc:499] Master 127.4.49.62:39957 was elected leader, sending a full tablet report...
I20250905 07:53:56.060281 10093 ts_tablet_manager.cc:1428] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:53:56.060281 10092 ts_tablet_manager.cc:1428] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:53:56.060400 10091 ts_tablet_manager.cc:1428] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:53:56.060624 10073 heartbeater.cc:499] Master 127.4.49.62:39957 was elected leader, sending a full tablet report...
I20250905 07:53:56.060624 9939 heartbeater.cc:499] Master 127.4.49.62:39957 was elected leader, sending a full tablet report...
W20250905 07:53:56.075954 9940 tablet.cc:2378] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:56.087689 10099 raft_consensus.cc:491] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:56.087898 10099 raft_consensus.cc:513] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.089151 10099 leader_election.cc:290] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35952083cc934170af7b23d9a4d87595 (127.4.49.2:37879), 3c9d9322981f4e8b95df9535665fc8b5 (127.4.49.3:37047)
I20250905 07:53:56.091250 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.089531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:53031 (local address 127.4.49.2:37879)
0905 07:53:56.089666 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.089670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.089689 (+ 19us) server_negotiation.cc:408] Connection header received
0905 07:53:56.089769 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.089772 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.089817 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.089912 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.090704 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.090706 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.090708 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.090720 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.090740 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.090747 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.090824 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.091014 (+ 190us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.091094 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:53:56.091260 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.089417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:37879 (local address 127.4.49.1:53031)
0905 07:53:56.089558 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.089574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.089678 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.089970 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.089974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.089990 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.090550 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.090553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.090569 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.090596 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.090832 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.090836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.090839 (+ 3us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.090926 (+ 87us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.091137 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20250905 07:53:56.091389 10098 raft_consensus.cc:491] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:56.091645 10098 raft_consensus.cc:513] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.092255 9893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35952083cc934170af7b23d9a4d87595" is_pre_election: true
I20250905 07:53:56.092336 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.089841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41907 (local address 127.4.49.3:37047)
0905 07:53:56.089994 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.089999 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.090235 (+ 236us) server_negotiation.cc:408] Connection header received
0905 07:53:56.090340 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.090343 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.090382 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.090447 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.091950 (+ 1503us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.091955 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.091958 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.091970 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.091990 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.091998 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.092104 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.092174 (+ 70us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.092215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250905 07:53:56.092453 9893 raft_consensus.cc:2466] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fb8aa2d41f407e8d35ccf0a3de500b in term 0.
I20250905 07:53:56.092706 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.089702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37047 (local address 127.4.49.1:41907)
0905 07:53:56.090165 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.090178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.090265 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.090480 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.090483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.090498 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.091757 (+ 1259us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.091760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.091775 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.091801 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.092099 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.092103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.092107 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.092180 (+ 73us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.092213 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":154,"threads_started":1}
I20250905 07:53:56.092975 9695 leader_election.cc:304] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35952083cc934170af7b23d9a4d87595, 36fb8aa2d41f407e8d35ccf0a3de500b; no voters:
I20250905 07:53:56.093029 10098 leader_election.cc:290] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fb8aa2d41f407e8d35ccf0a3de500b (127.4.49.1:44745), 3c9d9322981f4e8b95df9535665fc8b5 (127.4.49.3:37047)
I20250905 07:53:56.093328 10099 raft_consensus.cc:2802] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:53:56.093411 10099 raft_consensus.cc:491] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:53:56.093457 10099 raft_consensus.cc:3058] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:56.094827 10099 raft_consensus.cc:513] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.094841 10027 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9d9322981f4e8b95df9535665fc8b5" is_pre_election: true
I20250905 07:53:56.095155 10027 raft_consensus.cc:2466] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fb8aa2d41f407e8d35ccf0a3de500b in term 0.
I20250905 07:53:56.095301 10099 leader_election.cc:290] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [CANDIDATE]: Term 1 election: Requested vote from peers 35952083cc934170af7b23d9a4d87595 (127.4.49.2:37879), 3c9d9322981f4e8b95df9535665fc8b5 (127.4.49.3:37047)
I20250905 07:53:56.095696 9893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35952083cc934170af7b23d9a4d87595"
I20250905 07:53:56.095716 10027 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9d9322981f4e8b95df9535665fc8b5"
I20250905 07:53:56.095831 10027 raft_consensus.cc:3058] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:56.095831 9893 raft_consensus.cc:3058] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:56.095940 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.093351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:59385 (local address 127.4.49.1:44745)
0905 07:53:56.093531 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.093536 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.093943 (+ 407us) server_negotiation.cc:408] Connection header received
0905 07:53:56.094150 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.094153 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.094186 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.094264 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.095508 (+ 1244us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.095512 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.095514 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.095527 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.095550 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.095559 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.095618 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.095776 (+ 158us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.095820 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20250905 07:53:56.096271 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.093270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:44745 (local address 127.4.49.2:59385)
0905 07:53:56.093866 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.093881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.094006 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.094280 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.094284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.094299 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.095153 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.095155 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.095170 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.095196 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.095681 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.095684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.095686 (+ 2us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.096084 (+ 398us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.096125 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525}
I20250905 07:53:56.096498 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.093528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37047 (local address 127.4.49.2:56343)
0905 07:53:56.094640 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.094655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.094727 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.095037 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.095040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.095054 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:53:56.095650 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:53:56.095653 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:53:56.095680 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:53:56.095706 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:53:56.096223 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:53:56.096228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:53:56.096232 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:53:56.096309 (+ 77us) client_negotiation.cc:240] Negotiation successful
0905 07:53:56.096359 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":138,"threads_started":1}
I20250905 07:53:56.096710 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.093932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:56343 (local address 127.4.49.3:37047)
0905 07:53:56.094055 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.094059 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.094803 (+ 744us) server_negotiation.cc:408] Connection header received
0905 07:53:56.094846 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.094849 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.094885 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.094957 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.095844 (+ 887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.095847 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.095851 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.095863 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.095884 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.095893 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.095967 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.096553 (+ 586us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.096596 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:53:56.097139 9893 raft_consensus.cc:2466] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fb8aa2d41f407e8d35ccf0a3de500b in term 1.
I20250905 07:53:56.097570 10027 raft_consensus.cc:2466] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fb8aa2d41f407e8d35ccf0a3de500b in term 1.
I20250905 07:53:56.097657 10026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "35952083cc934170af7b23d9a4d87595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9d9322981f4e8b95df9535665fc8b5" is_pre_election: true
I20250905 07:53:56.097738 9695 leader_election.cc:304] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35952083cc934170af7b23d9a4d87595, 36fb8aa2d41f407e8d35ccf0a3de500b; no voters:
I20250905 07:53:56.097843 10026 raft_consensus.cc:2391] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35952083cc934170af7b23d9a4d87595 in current term 1: Already voted for candidate 36fb8aa2d41f407e8d35ccf0a3de500b in this term.
I20250905 07:53:56.097802 9760 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061650d7f2a548cbbafa23f7575cec50" candidate_uuid: "35952083cc934170af7b23d9a4d87595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" is_pre_election: true
I20250905 07:53:56.097906 10099 raft_consensus.cc:2802] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:56.098116 10099 raft_consensus.cc:695] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [term 1 LEADER]: Becoming Leader. State: Replica: 36fb8aa2d41f407e8d35ccf0a3de500b, State: Running, Role: LEADER
I20250905 07:53:56.098467 10099 consensus_queue.cc:237] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } }
I20250905 07:53:56.101092 9828 leader_election.cc:304] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35952083cc934170af7b23d9a4d87595; no voters: 36fb8aa2d41f407e8d35ccf0a3de500b, 3c9d9322981f4e8b95df9535665fc8b5
I20250905 07:53:56.101521 10098 raft_consensus.cc:2747] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:53:56.102164 9613 catalog_manager.cc:5582] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b reported cstate change: term changed from 0 to 1, leader changed from <none> to 36fb8aa2d41f407e8d35ccf0a3de500b (127.4.49.1). New cstate: current_term: 1 leader_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fb8aa2d41f407e8d35ccf0a3de500b" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 44745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20250905 07:53:56.121876 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.119340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.4.49.1:44745)
0905 07:53:56.119471 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.119475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.119488 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:53:56.119576 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.119579 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.119638 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.119813 (+ 175us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.121037 (+ 1224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.121041 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.121043 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.121055 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.121079 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.121087 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.121172 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.121468 (+ 296us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.121570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:53:56.127911 10026 raft_consensus.cc:1273] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 1 FOLLOWER]: Refusing update from remote peer 36fb8aa2d41f407e8d35ccf0a3de500b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:56.127940 9893 raft_consensus.cc:1273] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 1 FOLLOWER]: Refusing update from remote peer 36fb8aa2d41f407e8d35ccf0a3de500b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:53:56.128616 10099 consensus_queue.cc:1035] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9d9322981f4e8b95df9535665fc8b5" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:53:56.128876 10102 consensus_queue.cc:1035] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35952083cc934170af7b23d9a4d87595" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:53:56.128886 9808 tablet.cc:2378] T 061650d7f2a548cbbafa23f7575cec50 P 36fb8aa2d41f407e8d35ccf0a3de500b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:53:56.148931 9615 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42546:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:56.149112 9615 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42546:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:53:56.152096 9615 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 161179de6f4f4887a4c2e3e613973c48: Sending DeleteTablet for 3 replicas of tablet 061650d7f2a548cbbafa23f7575cec50
I20250905 07:53:56.153049 9740 tablet_service.cc:1515] Processing DeleteTablet for tablet 061650d7f2a548cbbafa23f7575cec50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:56 UTC) from {username='slave'} at 127.0.0.1:58238
I20250905 07:53:56.153020 9873 tablet_service.cc:1515] Processing DeleteTablet for tablet 061650d7f2a548cbbafa23f7575cec50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:56 UTC) from {username='slave'} at 127.0.0.1:59030
I20250905 07:53:56.153239 10006 tablet_service.cc:1515] Processing DeleteTablet for tablet 061650d7f2a548cbbafa23f7575cec50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:53:56 UTC) from {username='slave'} at 127.0.0.1:58226
I20250905 07:53:56.153836 10116 tablet_replica.cc:331] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: stopping tablet replica
I20250905 07:53:56.153836 10114 tablet_replica.cc:331] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: stopping tablet replica
I20250905 07:53:56.154143 10114 raft_consensus.cc:2241] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:56.154143 10116 raft_consensus.cc:2241] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:53:56.154510 10114 raft_consensus.cc:2270] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:56.154510 10116 raft_consensus.cc:2270] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:53:56.154748 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9677
I20250905 07:53:56.156215 10114 ts_tablet_manager.cc:1905] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:56.156215 10116 ts_tablet_manager.cc:1905] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:53:56.158910 10114 ts_tablet_manager.cc:1918] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:56.158910 10116 ts_tablet_manager.cc:1918] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:53:56.159031 10114 log.cc:1199] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/061650d7f2a548cbbafa23f7575cec50
I20250905 07:53:56.159030 10116 log.cc:1199] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/061650d7f2a548cbbafa23f7575cec50
I20250905 07:53:56.159322 10114 ts_tablet_manager.cc:1939] T 061650d7f2a548cbbafa23f7575cec50 P 35952083cc934170af7b23d9a4d87595: Deleting consensus metadata
I20250905 07:53:56.159341 10116 ts_tablet_manager.cc:1939] T 061650d7f2a548cbbafa23f7575cec50 P 3c9d9322981f4e8b95df9535665fc8b5: Deleting consensus metadata
I20250905 07:53:56.160805 9601 catalog_manager.cc:4928] TS 3c9d9322981f4e8b95df9535665fc8b5 (127.4.49.3:37047): tablet 061650d7f2a548cbbafa23f7575cec50 (table test-table [id=c18c00f1624043ec954644594ecbd458]) successfully deleted
I20250905 07:53:56.161018 9601 catalog_manager.cc:4928] TS 35952083cc934170af7b23d9a4d87595 (127.4.49.2:37879): tablet 061650d7f2a548cbbafa23f7575cec50 (table test-table [id=c18c00f1624043ec954644594ecbd458]) successfully deleted
W20250905 07:53:56.162811 9601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:44745 (error 108)
I20250905 07:53:56.162985 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9810
I20250905 07:53:56.164296 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.163985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:44745 (local address 127.0.0.1:58252)
0905 07:53:56.164148 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.164212 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:44745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20250905 07:53:56.164446 9601 catalog_manager.cc:4671] TS 36fb8aa2d41f407e8d35ccf0a3de500b (127.4.49.1:44745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 061650d7f2a548cbbafa23f7575cec50: Network error: Client connection negotiation failed: client connection to 127.4.49.1:44745: connect: Connection refused (error 111)
I20250905 07:53:56.169380 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9942
I20250905 07:53:56.174820 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 9583
2025-09-05T07:53:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (955 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20250905 07:53:56.197733 4292 mini_oidc.cc:150] Starting JWKS server
I20250905 07:53:56.199211 4292 webserver.cc:480] Webserver started at https://127.0.0.1:39123/ using document root <none> and password file <none>
I20250905 07:53:56.199364 4292 mini_oidc.cc:186] Starting OIDC Discovery server
I20250905 07:53:56.199626 4292 webserver.cc:480] Webserver started at http://127.0.0.1:41437/ using document root <none> and password file <none>
2025-09-05T07:53:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:53:56Z Disabled control of system clock
I20250905 07:53:56.208619 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45929
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:45929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/testchainca.cert with env {}
W20250905 07:53:56.324306 10123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:53:56.324595 10123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:53:56.324786 10123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:53:56.328437 10123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:53:56.328532 10123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:53:56.328553 10123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:53:56.328580 10123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:53:56.328603 10123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250905 07:53:56.328629 10123 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250905 07:53:56.328652 10123 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39123/jwks.json
I20250905 07:53:56.333649 10123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:45929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:53:56.334877 10123 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:53:56.336027 10123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:53:56.342633 10129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:53:56.342602 10131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:56.342801 10123 server_base.cc:1047] running on GCE node
W20250905 07:53:56.342713 10128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:53:56.343356 10123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:53:56.344458 10123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:53:56.345659 10123 hybrid_clock.cc:648] HybridClock initialized: now 1757058836345632 us; error 44 us; skew 500 ppm
I20250905 07:53:56.347832 10123 webserver.cc:480] Webserver started at http://127.4.49.62:36009/ using document root <none> and password file <none>
I20250905 07:53:56.348496 10123 fs_manager.cc:362] Metadata directory not provided
I20250905 07:53:56.348564 10123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:53:56.348739 10123 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:53:56.350675 10123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "460bb2f4a5244c06836303ada02ee306"
format_stamp: "Formatted at 2025-09-05 07:53:56 on dist-test-slave-mnfb"
server_key: "b7b6f37eaa4b2adbc7e0564d92f253c0"
server_key_iv: "b2f3e2d909ab697ec719f0d2ce8d0b03"
server_key_version: "encryptionkey@0"
I20250905 07:53:56.351177 10123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "460bb2f4a5244c06836303ada02ee306"
format_stamp: "Formatted at 2025-09-05 07:53:56 on dist-test-slave-mnfb"
server_key: "b7b6f37eaa4b2adbc7e0564d92f253c0"
server_key_iv: "b2f3e2d909ab697ec719f0d2ce8d0b03"
server_key_version: "encryptionkey@0"
I20250905 07:53:56.354754 10123 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250905 07:53:56.357257 10137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:53:56.358364 10123 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:53:56.358495 10123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "460bb2f4a5244c06836303ada02ee306"
format_stamp: "Formatted at 2025-09-05 07:53:56 on dist-test-slave-mnfb"
server_key: "b7b6f37eaa4b2adbc7e0564d92f253c0"
server_key_iv: "b2f3e2d909ab697ec719f0d2ce8d0b03"
server_key_version: "encryptionkey@0"
I20250905 07:53:56.358590 10123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:53:56.368160 10123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:53:56.368801 10123 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:53:56.368952 10123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:53:56.375732 10123 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:45929
I20250905 07:53:56.375737 10190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:45929 every 8 connection(s)
I20250905 07:53:56.376842 10123 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:53:56.379813 10191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:53:56.384824 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10123
I20250905 07:53:56.384933 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:53:56.385141 4292 external_mini_cluster.cc:1468] Setting key 9d9cd954806100f1edca7c67b8d879ea
I20250905 07:53:56.385708 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: Bootstrap starting.
I20250905 07:53:56.387995 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: Neither blocks nor log segments found. Creating new log.
I20250905 07:53:56.388935 10191 log.cc:826] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: Log is configured to *not* fsync() on all Append() calls
I20250905 07:53:56.391052 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: No bootstrap required, opened a new log
I20250905 07:53:56.393631 10191 raft_consensus.cc:357] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } }
I20250905 07:53:56.393846 10191 raft_consensus.cc:383] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:53:56.393906 10191 raft_consensus.cc:738] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460bb2f4a5244c06836303ada02ee306, State: Initialized, Role: FOLLOWER
I20250905 07:53:56.394325 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } }
I20250905 07:53:56.394454 10191 raft_consensus.cc:397] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:53:56.394513 10191 raft_consensus.cc:491] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:53:56.394658 10191 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:53:56.395649 10191 raft_consensus.cc:513] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } }
I20250905 07:53:56.395999 10191 leader_election.cc:304] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460bb2f4a5244c06836303ada02ee306; no voters:
I20250905 07:53:56.396342 10191 leader_election.cc:290] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:53:56.396431 10196 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:53:56.396593 10196 raft_consensus.cc:695] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [term 1 LEADER]: Becoming Leader. State: Replica: 460bb2f4a5244c06836303ada02ee306, State: Running, Role: LEADER
I20250905 07:53:56.396898 10196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } }
I20250905 07:53:56.397027 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.386401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.4.49.62:45929)
0905 07:53:56.387046 (+ 645us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.387055 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.387093 (+ 38us) server_negotiation.cc:408] Connection header received
0905 07:53:56.387656 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.387675 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.388041 (+ 366us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.388382 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:53:56.388930 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.389894 (+ 964us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:56.390732 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.390938 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:56.395148 (+ 4210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:53:56.395175 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:53:56.395188 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:53:56.395599 (+ 411us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:53:56.395648 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:53:56.395682 (+ 34us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:53:56.395797 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.395964 (+ 167us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.396260 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
I20250905 07:53:56.397603 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:53:56.400224 4292 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250905 07:53:56.399921 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460bb2f4a5244c06836303ada02ee306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } } }
I20250905 07:53:56.400400 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [sys.catalog]: This master's current role is: LEADER
I20250905 07:53:56.400045 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460bb2f4a5244c06836303ada02ee306. Latest consensus state: current_term: 1 leader_uuid: "460bb2f4a5244c06836303ada02ee306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460bb2f4a5244c06836303ada02ee306" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45929 } } }
I20250905 07:53:56.400616 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306 [sys.catalog]: This master's current role is: LEADER
W20250905 07:53:56.402765 10212 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:53:56.402863 10212 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:53:56.402962 10213 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:53:56.403879 10213 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:53:56.409895 10213 catalog_manager.cc:1349] Generated new cluster ID: 0f9283c3dcac4495833a17a17e24ca9e
I20250905 07:53:56.409987 10213 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:53:56.457885 10213 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:53:56.459215 10213 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:53:56.468043 10213 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 460bb2f4a5244c06836303ada02ee306: Generated new TSK 0
I20250905 07:53:56.468844 10213 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:53:56.516788 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.486433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.4.49.62:45929)
0905 07:53:56.486580 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.486583 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.486764 (+ 181us) server_negotiation.cc:408] Connection header received
0905 07:53:56.486990 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.486993 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.487055 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.487141 (+ 86us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:53:56.487588 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.488181 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:56.489122 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.489267 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:56.489367 (+ 100us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:53:56.516185 (+ 26818us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0905 07:53:56.516288 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:53:56.516534 (+ 246us) server_negotiation.cc:300] Negotiation successful
0905 07:53:56.516602 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":150,"threads_started":1}
I20250905 07:53:56.528203 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:53:56.524586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.4.49.62:45929)
0905 07:53:56.524733 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:53:56.524737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:53:56.524809 (+ 72us) server_negotiation.cc:408] Connection header received
0905 07:53:56.524940 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:53:56.524943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:53:56.524989 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:53:56.525055 (+ 66us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:53:56.525468 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.525967 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:53:56.526851 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.526982 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:56.527026 (+ 44us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:53:56.527884 (+ 858us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0905 07:53:56.528097 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36784: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":73}
W20250905 07:53:56.528277 10194 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36784: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20250905 07:53:56.529107 10228 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:53:56.524449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45929 (local address 127.0.0.1:36784)
0905 07:53:56.524725 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0905 07:53:56.524741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:53:56.524854 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:53:56.525074 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:53:56.525077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:53:56.525085 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0905 07:53:56.525368 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:56.525376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.526084 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:53:56.526089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:53:56.526747 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:53:56.526758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:53:56.526865 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:53:56.526890 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0905 07:53:56.528101 (+ 1211us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0905 07:53:56.528276 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.49.62:45929: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20250905 07:54:11.537325 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:11.534227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.4.49.62:45929)
0905 07:54:11.534503 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:11.534506 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:11.534522 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:11.534596 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:11.534599 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:11.534646 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:11.534731 (+ 85us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:54:11.535069 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.535598 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:11.536236 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.536366 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:11.536411 (+ 45us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:54:11.537046 (+ 635us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0905 07:54:11.537201 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47424: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
W20250905 07:54:11.537420 10234 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47424: JWT verification failed: token expired
W20250905 07:54:11.537415 10233 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:11.534089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45929 (local address 127.0.0.1:47424)
0905 07:54:11.534398 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:11.534413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:11.534514 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:11.534750 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:11.534752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:11.534759 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0905 07:54:11.534976 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:11.534982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.535695 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:11.535698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:11.536144 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:11.536152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.536254 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:11.536278 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0905 07:54:11.537150 (+ 872us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0905 07:54:11.537269 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.49.62:45929: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
W20250905 07:54:11.539944 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:11.539149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.4.49.62:45929)
0905 07:54:11.539280 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:11.539285 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:11.539353 (+ 68us) server_negotiation.cc:408] Connection header received
0905 07:54:11.539501 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:11.539505 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:11.539551 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:11.539618 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:11.539826 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47426: BlockingRecv error: recv got EOF from 127.0.0.1:47426 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20250905 07:54:11.543262 10244 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.4.49.62:45929
W20250905 07:54:11.543841 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:11.542990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.4.49.62:45929)
0905 07:54:11.543118 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:11.543123 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:11.543198 (+ 75us) server_negotiation.cc:408] Connection header received
0905 07:54:11.543430 (+ 232us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:11.543433 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:11.543495 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:11.543560 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:11.543745 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47442: BlockingRecv error: recv got EOF from 127.0.0.1:47442 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:54:11.544198 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10123
2025-09-05T07:54:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15386 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20250905 07:54:11.583565 4292 mini_oidc.cc:150] Starting JWKS server
I20250905 07:54:11.584554 4292 webserver.cc:480] Webserver started at https://127.0.0.1:33311/ using document root <none> and password file <none>
I20250905 07:54:11.584640 4292 mini_oidc.cc:186] Starting OIDC Discovery server
I20250905 07:54:11.584841 4292 webserver.cc:480] Webserver started at http://127.0.0.1:44047/ using document root <none> and password file <none>
2025-09-05T07:54:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:11Z Disabled control of system clock
I20250905 07:54:11.592833 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45323
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:45323
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/testchainca.cert with env {}
W20250905 07:54:11.701541 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:11.701793 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:11.701845 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:11.705238 10250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:11.705327 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:11.705341 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:11.705360 10250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:11.705376 10250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250905 07:54:11.705406 10250 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250905 07:54:11.705420 10250 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33311/jwks.json
I20250905 07:54:11.710075 10250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:45323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:11.711190 10250 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:11.712195 10250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:11.718147 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:11.718176 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:11.718147 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:11.718273 10250 server_base.cc:1047] running on GCE node
I20250905 07:54:11.718935 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:11.719914 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:11.721081 10250 hybrid_clock.cc:648] HybridClock initialized: now 1757058851721062 us; error 41 us; skew 500 ppm
I20250905 07:54:11.723017 10250 webserver.cc:480] Webserver started at http://127.4.49.62:43817/ using document root <none> and password file <none>
I20250905 07:54:11.723609 10250 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:11.723672 10250 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:11.723850 10250 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:11.725713 10250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "12e71bb206c64c2fac20c738f664d4da"
format_stamp: "Formatted at 2025-09-05 07:54:11 on dist-test-slave-mnfb"
server_key: "50a5120db0f747fa11c5130f0e372708"
server_key_iv: "6dca5ec424d6f157786f7375d2802164"
server_key_version: "encryptionkey@0"
I20250905 07:54:11.726208 10250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "12e71bb206c64c2fac20c738f664d4da"
format_stamp: "Formatted at 2025-09-05 07:54:11 on dist-test-slave-mnfb"
server_key: "50a5120db0f747fa11c5130f0e372708"
server_key_iv: "6dca5ec424d6f157786f7375d2802164"
server_key_version: "encryptionkey@0"
I20250905 07:54:11.729825 10250 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:11.732185 10265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:11.733278 10250 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:11.733408 10250 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "12e71bb206c64c2fac20c738f664d4da"
format_stamp: "Formatted at 2025-09-05 07:54:11 on dist-test-slave-mnfb"
server_key: "50a5120db0f747fa11c5130f0e372708"
server_key_iv: "6dca5ec424d6f157786f7375d2802164"
server_key_version: "encryptionkey@0"
I20250905 07:54:11.733523 10250 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:11.751228 10250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:11.751952 10250 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:11.752164 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:11.760367 10250 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:45323
I20250905 07:54:11.760367 10318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:45323 every 8 connection(s)
I20250905 07:54:11.761456 10250 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:11.764398 10319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:11.768332 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10250
I20250905 07:54:11.768446 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:11.768666 4292 external_mini_cluster.cc:1468] Setting key 7a8f38279add6dd03bef3925241d0d22
I20250905 07:54:11.770506 10319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da: Bootstrap starting.
I20250905 07:54:11.773422 10319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:11.774351 10319 log.cc:826] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:11.776693 10319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da: No bootstrap required, opened a new log
I20250905 07:54:11.777604 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:11.769933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.4.49.62:45323)
0905 07:54:11.770667 (+ 734us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:11.770678 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:11.770725 (+ 47us) server_negotiation.cc:408] Connection header received
0905 07:54:11.771427 (+ 702us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:11.771463 (+ 36us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:11.771948 (+ 485us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:11.772377 (+ 429us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:11.772883 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.774008 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:11.774773 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.774988 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:11.775653 (+ 665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:11.775680 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:11.775702 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:11.776201 (+ 499us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:11.776257 (+ 56us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:11.776275 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:11.776386 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:11.776557 (+ 171us) server_negotiation.cc:300] Negotiation successful
0905 07:54:11.776823 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20250905 07:54:11.779521 10319 raft_consensus.cc:357] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } }
I20250905 07:54:11.779788 10319 raft_consensus.cc:383] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:11.779844 10319 raft_consensus.cc:738] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e71bb206c64c2fac20c738f664d4da, State: Initialized, Role: FOLLOWER
I20250905 07:54:11.780321 10319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } }
I20250905 07:54:11.780473 10319 raft_consensus.cc:397] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:11.780541 10319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:11.780623 4292 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250905 07:54:11.780658 10319 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:11.781631 10319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } }
I20250905 07:54:11.782047 10319 leader_election.cc:304] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e71bb206c64c2fac20c738f664d4da; no voters:
I20250905 07:54:11.782433 10319 leader_election.cc:290] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:11.782541 10324 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:11.782804 10324 raft_consensus.cc:695] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [term 1 LEADER]: Becoming Leader. State: Replica: 12e71bb206c64c2fac20c738f664d4da, State: Running, Role: LEADER
I20250905 07:54:11.783142 10324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } }
I20250905 07:54:11.783643 10319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:11.784911 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e71bb206c64c2fac20c738f664d4da. Latest consensus state: current_term: 1 leader_uuid: "12e71bb206c64c2fac20c738f664d4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } } }
I20250905 07:54:11.784921 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e71bb206c64c2fac20c738f664d4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e71bb206c64c2fac20c738f664d4da" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 45323 } } }
I20250905 07:54:11.785238 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:11.785218 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:11.785702 10333 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:11.789085 10333 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:11.795166 10333 catalog_manager.cc:1349] Generated new cluster ID: 5d10562ce25947b1bbf32653e0e89909
I20250905 07:54:11.795259 10333 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:11.803426 10333 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:11.804769 10333 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:11.821389 10333 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 12e71bb206c64c2fac20c738f664d4da: Generated new TSK 0
I20250905 07:54:11.822372 10333 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250905 07:54:11.849138 10349 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250905 07:54:11.849221 10349 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250905 07:54:11.849957 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:11.832480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.4.49.62:45323)
0905 07:54:11.832627 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:11.832640 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:11.832719 (+ 79us) server_negotiation.cc:408] Connection header received
0905 07:54:11.832917 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:11.832922 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:11.832996 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:11.833121 (+ 125us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:54:11.833509 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.834374 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:11.835130 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.835341 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:11.835479 (+ 138us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:54:11.849508 (+ 14029us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0905 07:54:11.849809 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20250905 07:54:11.850044 10322 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250905 07:54:11.850064 10348 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:11.832326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:45323 (local address 127.0.0.1:36462)
0905 07:54:11.832631 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:11.832644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:11.832758 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:11.833147 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:11.833150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:11.833158 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0905 07:54:11.833390 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:11.833397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.834553 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:11.834557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:11.835020 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:11.835030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:11.835132 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:11.835156 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0905 07:54:11.849763 (+ 14607us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0905 07:54:11.849899 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.49.62:45323: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":129,"threads_started":1}
I20250905 07:54:11.850855 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10250
2025-09-05T07:54:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20250905 07:54:11.889990 4292 mini_oidc.cc:150] Starting JWKS server
I20250905 07:54:11.890903 4292 webserver.cc:480] Webserver started at https://127.0.0.1:42365/ using document root <none> and password file <none>
I20250905 07:54:11.890997 4292 mini_oidc.cc:186] Starting OIDC Discovery server
I20250905 07:54:11.891211 4292 webserver.cc:480] Webserver started at http://127.0.0.1:33481/ using document root <none> and password file <none>
2025-09-05T07:54:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:11Z Disabled control of system clock
I20250905 07:54:11.898597 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39717
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:39717
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250905 07:54:12.006974 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:12.007246 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:12.007318 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:12.010740 10356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:12.010833 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:12.010856 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:12.010882 10356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:12.010907 10356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250905 07:54:12.010938 10356 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250905 07:54:12.010965 10356 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42365/jwks.json
I20250905 07:54:12.015039 10356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:39717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:12.016251 10356 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:12.017467 10356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 07:54:12.023165 10356 server_base.cc:1047] running on GCE node
W20250905 07:54:12.023058 10361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:12.023190 10364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:12.023058 10362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:12.023908 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:12.024884 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:12.026042 10356 hybrid_clock.cc:648] HybridClock initialized: now 1757058852026017 us; error 46 us; skew 500 ppm
I20250905 07:54:12.027935 10356 webserver.cc:480] Webserver started at http://127.4.49.62:45331/ using document root <none> and password file <none>
I20250905 07:54:12.028543 10356 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:12.028604 10356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:12.028806 10356 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:12.030628 10356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "62dd54ef8c6e4eb09f2e262843d0b36b"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "e549b811ac6956d11128303809ec2761"
server_key_iv: "e995d1b554da19051b805b238e9f93fb"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.031116 10356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "62dd54ef8c6e4eb09f2e262843d0b36b"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "e549b811ac6956d11128303809ec2761"
server_key_iv: "e995d1b554da19051b805b238e9f93fb"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.034711 10356 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250905 07:54:12.037122 10370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:12.038246 10356 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:12.038368 10356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "62dd54ef8c6e4eb09f2e262843d0b36b"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "e549b811ac6956d11128303809ec2761"
server_key_iv: "e995d1b554da19051b805b238e9f93fb"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.038472 10356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:12.056496 10356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:12.057222 10356 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:12.057427 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:12.064584 10422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:39717 every 8 connection(s)
I20250905 07:54:12.064584 10356 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:39717
I20250905 07:54:12.065673 10356 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:12.068596 10423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:12.073997 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10356
I20250905 07:54:12.074110 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:12.074316 4292 external_mini_cluster.cc:1468] Setting key cf63923b86437cfb3b021a1223c60d4b
I20250905 07:54:12.074321 10423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b: Bootstrap starting.
I20250905 07:54:12.076792 10423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:12.077596 10423 log.cc:826] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:12.079789 10423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b: No bootstrap required, opened a new log
I20250905 07:54:12.082388 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:12.075528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.4.49.62:39717)
0905 07:54:12.076196 (+ 668us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:12.076205 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:12.076243 (+ 38us) server_negotiation.cc:408] Connection header received
0905 07:54:12.076829 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:12.076849 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:12.077244 (+ 395us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:12.077566 (+ 322us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:12.078151 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.079029 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.079908 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.080186 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.080706 (+ 520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:12.080727 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:12.080738 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:12.081113 (+ 375us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:12.081155 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:12.081170 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:12.081254 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:12.081487 (+ 233us) server_negotiation.cc:300] Negotiation successful
0905 07:54:12.081763 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20250905 07:54:12.082681 10423 raft_consensus.cc:357] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } }
I20250905 07:54:12.082890 10423 raft_consensus.cc:383] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:12.082957 10423 raft_consensus.cc:738] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd54ef8c6e4eb09f2e262843d0b36b, State: Initialized, Role: FOLLOWER
I20250905 07:54:12.083367 10423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } }
I20250905 07:54:12.083513 10423 raft_consensus.cc:397] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:12.083573 10423 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:12.083693 10423 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:12.084782 10423 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } }
I20250905 07:54:12.085172 10423 leader_election.cc:304] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62dd54ef8c6e4eb09f2e262843d0b36b; no voters:
I20250905 07:54:12.085455 4292 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250905 07:54:12.085546 10423 leader_election.cc:290] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:12.085655 10428 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:12.085870 10428 raft_consensus.cc:695] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [term 1 LEADER]: Becoming Leader. State: Replica: 62dd54ef8c6e4eb09f2e262843d0b36b, State: Running, Role: LEADER
I20250905 07:54:12.086186 10428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } }
I20250905 07:54:12.086942 10423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:12.087548 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd54ef8c6e4eb09f2e262843d0b36b. Latest consensus state: current_term: 1 leader_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } } }
I20250905 07:54:12.087668 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:12.088294 10429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd54ef8c6e4eb09f2e262843d0b36b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39717 } } }
I20250905 07:54:12.088443 10429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:12.088912 10432 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:12.092489 10432 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:12.097949 10432 catalog_manager.cc:1349] Generated new cluster ID: e924ee5877f14c248bd9378ca8c82014
I20250905 07:54:12.098038 10432 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:12.102829 10432 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:12.103854 10432 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:12.110181 10432 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 62dd54ef8c6e4eb09f2e262843d0b36b: Generated new TSK 0
I20250905 07:54:12.110954 10432 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250905 07:54:12.137682 10454 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250905 07:54:12.137761 10454 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20250905 07:54:12.139523 10453 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:12.119026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39717 (local address 127.0.0.1:41904)
0905 07:54:12.119288 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:12.119300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:12.119398 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:12.119789 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:12.119792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:12.119801 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0905 07:54:12.120030 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:12.120037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.121174 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.121177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:12.121606 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:12.121615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.121713 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.121735 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0905 07:54:12.139203 (+ 17468us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0905 07:54:12.139327 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.49.62:39717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20250905 07:54:12.139523 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:12.119149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.4.49.62:39717)
0905 07:54:12.119292 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:12.119297 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:12.119365 (+ 68us) server_negotiation.cc:408] Connection header received
0905 07:54:12.119543 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:12.119548 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:12.119616 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:12.119743 (+ 127us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:54:12.120199 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.121032 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.121721 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.121929 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.122054 (+ 125us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:54:12.138987 (+ 16933us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0905 07:54:12.139379 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20250905 07:54:12.139585 10426 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20250905 07:54:12.140267 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10356
2025-09-05T07:54:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (289 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-09-05T07:54:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:12Z Disabled control of system clock
I20250905 07:54:12.187335 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:32823
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:32823
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/testchainca.cert with env {}
W20250905 07:54:12.295079 10458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:12.295332 10458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:12.295383 10458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:12.298765 10458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:12.298849 10458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:12.298866 10458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:12.298884 10458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:12.298906 10458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250905 07:54:12.298923 10458 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250905 07:54:12.298941 10458 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20250905 07:54:12.303555 10458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:32823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:12.304675 10458 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:12.305490 10458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:12.310722 10466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:12.310730 10464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:12.310770 10463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:12.311189 10458 server_base.cc:1047] running on GCE node
I20250905 07:54:12.311702 10458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:12.312674 10458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:12.313849 10458 hybrid_clock.cc:648] HybridClock initialized: now 1757058852313831 us; error 35 us; skew 500 ppm
I20250905 07:54:12.315616 10458 webserver.cc:480] Webserver started at http://127.4.49.62:34917/ using document root <none> and password file <none>
I20250905 07:54:12.316226 10458 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:12.316289 10458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:12.316481 10458 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:12.318254 10458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "0dada8f35cd51c2df11fffc46c7266c0"
server_key_iv: "e03f4870fbc04065805a44a0a1a3fabf"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.318727 10458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "0dada8f35cd51c2df11fffc46c7266c0"
server_key_iv: "e03f4870fbc04065805a44a0a1a3fabf"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.322115 10458 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250905 07:54:12.324405 10472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:12.325485 10458 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 07:54:12.325618 10458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f"
format_stamp: "Formatted at 2025-09-05 07:54:12 on dist-test-slave-mnfb"
server_key: "0dada8f35cd51c2df11fffc46c7266c0"
server_key_iv: "e03f4870fbc04065805a44a0a1a3fabf"
server_key_version: "encryptionkey@0"
I20250905 07:54:12.325731 10458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:12.342541 10458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:12.343245 10458 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:12.343425 10458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:12.350472 10458 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:32823
I20250905 07:54:12.350521 10525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:32823 every 8 connection(s)
I20250905 07:54:12.351497 10458 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:12.352646 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10458
I20250905 07:54:12.352769 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:12.353032 4292 external_mini_cluster.cc:1468] Setting key 278782d976ff3607db35d5ee46584cea
I20250905 07:54:12.354859 10526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:12.360019 10526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f: Bootstrap starting.
I20250905 07:54:12.361178 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:12.354222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.4.49.62:32823)
0905 07:54:12.354876 (+ 654us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:12.354885 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:12.354924 (+ 39us) server_negotiation.cc:408] Connection header received
0905 07:54:12.355585 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:12.355603 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:12.355963 (+ 360us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:12.356322 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:12.356858 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.358029 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.358717 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.358999 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.359572 (+ 573us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:12.359615 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:12.359627 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:12.359987 (+ 360us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:12.360028 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:12.360043 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:12.360180 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:12.360334 (+ 154us) server_negotiation.cc:300] Negotiation successful
0905 07:54:12.360591 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20250905 07:54:12.362425 10526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:12.363204 10526 log.cc:826] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:12.363945 4292 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250905 07:54:12.365060 10526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f: No bootstrap required, opened a new log
I20250905 07:54:12.367534 10526 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } }
I20250905 07:54:12.367743 10526 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:12.367813 10526 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d66fae96d1d4a96bc953c62ec5a9c7f, State: Initialized, Role: FOLLOWER
I20250905 07:54:12.368255 10526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } }
I20250905 07:54:12.368384 10526 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:12.368446 10526 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:12.368551 10526 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:12.369400 10526 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } }
I20250905 07:54:12.369731 10526 leader_election.cc:304] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d66fae96d1d4a96bc953c62ec5a9c7f; no voters:
I20250905 07:54:12.370023 10526 leader_election.cc:290] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:12.370133 10531 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:12.370342 10531 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [term 1 LEADER]: Becoming Leader. State: Replica: 7d66fae96d1d4a96bc953c62ec5a9c7f, State: Running, Role: LEADER
I20250905 07:54:12.370669 10531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } }
I20250905 07:54:12.371204 10526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:12.372313 10532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } } }
I20250905 07:54:12.372311 10533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d66fae96d1d4a96bc953c62ec5a9c7f. Latest consensus state: current_term: 1 leader_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d66fae96d1d4a96bc953c62ec5a9c7f" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 32823 } } }
I20250905 07:54:12.372457 10532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:12.372459 10533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:12.372856 10540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:12.375936 10540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:12.381618 10540 catalog_manager.cc:1349] Generated new cluster ID: 0d6c9b3dc07e4bf0a547da226b9dc646
I20250905 07:54:12.381727 10540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:12.391762 10540 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:12.392721 10540 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:12.406534 10540 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7d66fae96d1d4a96bc953c62ec5a9c7f: Generated new TSK 0
I20250905 07:54:12.407352 10540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:54:12.468190 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:12.415471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.4.49.62:32823)
0905 07:54:12.415631 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:12.415634 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:12.415755 (+ 121us) server_negotiation.cc:408] Connection header received
0905 07:54:12.415932 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:12.415934 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:12.415993 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:12.416110 (+ 117us) server_negotiation.cc:227] Negotiated authn=JWT
0905 07:54:12.416574 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.417401 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.418335 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.418501 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.418606 (+ 105us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0905 07:54:12.467765 (+ 49159us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0905 07:54:12.468029 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33176: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20250905 07:54:12.468267 10529 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33176: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20250905 07:54:12.468394 10555 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:12.415326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:32823 (local address 127.0.0.1:33176)
0905 07:54:12.415662 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:12.415678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:12.415800 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:12.416119 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:12.416123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:12.416130 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0905 07:54:12.416421 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:12.416431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.417531 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:12.417536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:12.418198 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:12.418212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:12.418339 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:12.418368 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0905 07:54:12.467993 (+ 49625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0905 07:54:12.468212 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.49.62:32823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":143,"threads_started":1}
I20250905 07:54:12.468948 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10458
2025-09-05T07:54:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (307 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:12 dist-test-slave-mnfb krb5kdc[10567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:12 dist-test-slave-mnfb krb5kdc[10567](info): set up 2 sockets
Sep 05 07:54:12 dist-test-slave-mnfb krb5kdc[10567](info): commencing operation
krb5kdc: starting...
W20250905 07:54:14.535374 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.813s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:14 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:14.671243 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36695
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:14.787004 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:14.787261 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:14.787328 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:14.790824 10584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:14.790917 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:14.790940 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:14.790958 10584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:14.790975 10584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:14.795637 10584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:36695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:14.796926 10584 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:14.797890 10584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:14.803885 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:14.803895 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:14.803875 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:14.804226 10584 server_base.cc:1047] running on GCE node
I20250905 07:54:14.804787 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:14.805773 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:14.806931 10584 hybrid_clock.cc:648] HybridClock initialized: now 1757058854806912 us; error 37 us; skew 500 ppm
Sep 05 07:54:14 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:14.809927 10584 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:54:14.811136 10584 webserver.cc:480] Webserver started at http://127.4.49.62:36297/ using document root <none> and password file <none>
I20250905 07:54:14.811713 10584 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:14.811764 10584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:14.811939 10584 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:14.813843 10584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "413677d88f4f44938e6a8013a894bedb"
format_stamp: "Formatted at 2025-09-05 07:54:14 on dist-test-slave-mnfb"
server_key: "92e4a8dc7770757ab3d1235b21098968"
server_key_iv: "9e1feabeb29b3accf116c00d3cfa7e9a"
server_key_version: "encryptionkey@0"
I20250905 07:54:14.814316 10584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "413677d88f4f44938e6a8013a894bedb"
format_stamp: "Formatted at 2025-09-05 07:54:14 on dist-test-slave-mnfb"
server_key: "92e4a8dc7770757ab3d1235b21098968"
server_key_iv: "9e1feabeb29b3accf116c00d3cfa7e9a"
server_key_version: "encryptionkey@0"
I20250905 07:54:14.818027 10584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:14.820619 10599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:14.821887 10584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:14.822045 10584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "413677d88f4f44938e6a8013a894bedb"
format_stamp: "Formatted at 2025-09-05 07:54:14 on dist-test-slave-mnfb"
server_key: "92e4a8dc7770757ab3d1235b21098968"
server_key_iv: "9e1feabeb29b3accf116c00d3cfa7e9a"
server_key_version: "encryptionkey@0"
I20250905 07:54:14.822166 10584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:14.835093 10584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:14.841048 10584 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:14.841248 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:14.848767 10652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:36695 every 8 connection(s)
I20250905 07:54:14.848768 10584 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:36695
I20250905 07:54:14.849960 10584 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:14.853132 10653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:14.857860 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10584
I20250905 07:54:14.857978 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:14.858198 4292 external_mini_cluster.cc:1468] Setting key b8ce82f65d5a5f5099fb09710b23a342
I20250905 07:54:14.859880 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: Bootstrap starting.
I20250905 07:54:14.862879 10653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:14.863894 10653 log.cc:826] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:14.866266 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: No bootstrap required, opened a new log
Sep 05 07:54:14 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:14.869624 10653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } }
I20250905 07:54:14.869897 10653 raft_consensus.cc:383] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:14.869954 10653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413677d88f4f44938e6a8013a894bedb, State: Initialized, Role: FOLLOWER
I20250905 07:54:14.870474 10653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } }
I20250905 07:54:14.870646 10653 raft_consensus.cc:397] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:14.870702 10653 raft_consensus.cc:491] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:14.870844 10653 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:14.871996 10653 raft_consensus.cc:513] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } }
I20250905 07:54:14.872407 10653 leader_election.cc:304] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 413677d88f4f44938e6a8013a894bedb; no voters:
I20250905 07:54:14.872710 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:14.859661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.4.49.62:36695)
0905 07:54:14.860193 (+ 532us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:14.860206 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:14.860251 (+ 45us) server_negotiation.cc:408] Connection header received
0905 07:54:14.861048 (+ 797us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:14.861082 (+ 34us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:14.861555 (+ 473us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:14.861983 (+ 428us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:14.862870 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:14.864007 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:14.864882 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:14.865186 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:14.867316 (+ 2130us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:14.867343 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:14.867359 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:14.867409 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:14.869540 (+ 2131us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:14.870062 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:14.870069 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:14.870074 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:14.870253 (+ 179us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:14.870617 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:14.870622 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:14.870624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:14.871102 (+ 478us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:14.871300 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:14.871636 (+ 336us) server_negotiation.cc:300] Negotiation successful
0905 07:54:14.871937 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20250905 07:54:14.872771 10653 leader_election.cc:290] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:14.872881 10658 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:14.873090 10658 raft_consensus.cc:695] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [term 1 LEADER]: Becoming Leader. State: Replica: 413677d88f4f44938e6a8013a894bedb, State: Running, Role: LEADER
I20250905 07:54:14.873476 10658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } }
I20250905 07:54:14.874035 10653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:14.876502 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "413677d88f4f44938e6a8013a894bedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } } }
I20250905 07:54:14.876670 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:14.877512 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 413677d88f4f44938e6a8013a894bedb. Latest consensus state: current_term: 1 leader_uuid: "413677d88f4f44938e6a8013a894bedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413677d88f4f44938e6a8013a894bedb" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36695 } } }
I20250905 07:54:14.877617 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb [sys.catalog]: This master's current role is: LEADER
W20250905 07:54:14.879799 10672 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:54:14.879884 10672 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:54:14.879998 10673 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:14.880812 10673 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:14.886487 10673 catalog_manager.cc:1349] Generated new cluster ID: e482bffb7ec94f67b5a4cb5636d390c3
I20250905 07:54:14.886584 10673 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:14.907332 10673 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:14.908776 10673 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:14.920982 10673 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 413677d88f4f44938e6a8013a894bedb: Generated new TSK 0
I20250905 07:54:14.921983 10673 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:14.969329 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36695
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:15.081598 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:15.081861 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:15.081923 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:15.085422 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:15.085511 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:15.085600 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:15.090229 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:15.091434 10681 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:15.092373 10681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:15.099507 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:15.099494 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:15.099498 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:15.099676 10681 server_base.cc:1047] running on GCE node
I20250905 07:54:15.100275 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:15.100914 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:15.102068 10681 hybrid_clock.cc:648] HybridClock initialized: now 1757058855102052 us; error 31 us; skew 500 ppm
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.105087 10681 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:54:15.106287 10681 webserver.cc:480] Webserver started at http://127.4.49.1:37307/ using document root <none> and password file <none>
I20250905 07:54:15.106894 10681 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:15.106949 10681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:15.107151 10681 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:15.109143 10681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "fbcc842e26d44f7f8dfc24288a3846a5"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "8bab7729e99d544aee85d998242ff954"
server_key_iv: "e4cfb025bb3c49e568ed867b1037b79b"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.109706 10681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "fbcc842e26d44f7f8dfc24288a3846a5"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "8bab7729e99d544aee85d998242ff954"
server_key_iv: "e4cfb025bb3c49e568ed867b1037b79b"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.113385 10681 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:15.115839 10696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.117039 10681 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:15.117172 10681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "fbcc842e26d44f7f8dfc24288a3846a5"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "8bab7729e99d544aee85d998242ff954"
server_key_iv: "e4cfb025bb3c49e568ed867b1037b79b"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.117288 10681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:15.138085 10681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:15.141212 10681 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:15.141414 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:15.142109 10681 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:15.143023 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:15.143074 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.143131 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:15.143158 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.152047 10681 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:41059
I20250905 07:54:15.152114 10812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:41059 every 8 connection(s)
I20250905 07:54:15.153133 10681 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:15.155938 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10681
I20250905 07:54:15.156102 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:15.156373 4292 external_mini_cluster.cc:1468] Setting key a1815d03c3b77e60c4aff3b20e05d37e
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:54:15.166529 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.155114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:43757 (local address 127.4.49.62:36695)
0905 07:54:15.155291 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.155295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.156033 (+ 738us) server_negotiation.cc:408] Connection header received
0905 07:54:15.157015 (+ 982us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.157019 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.157076 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.157152 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.158626 (+ 1474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.159151 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.160358 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.160553 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.162983 (+ 2430us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.163005 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.163008 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.163038 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.164578 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.165091 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.165095 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.165097 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.165149 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.165425 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.165428 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.165430 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.165628 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.165736 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.166248 (+ 512us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.166372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:54:15.167339 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.155400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36695 (local address 127.4.49.1:43757)
0905 07:54:15.155880 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:15.155919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:15.156802 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:15.157339 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:15.157349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:15.157883 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:15.158479 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.158493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.159267 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.159271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:15.160219 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.160230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.160462 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.161081 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:15.161109 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:15.162772 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:15.164712 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.164719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.164734 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.164988 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.165233 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.165236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.165238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.165339 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.165745 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:15.165752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:15.166013 (+ 261us) client_negotiation.cc:769] Sending connection context
0905 07:54:15.166208 (+ 195us) client_negotiation.cc:240] Negotiation successful
0905 07:54:15.166480 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":133,"threads_started":1}
I20250905 07:54:15.168747 10813 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36695
I20250905 07:54:15.169055 10813 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:15.169716 10813 heartbeater.cc:507] Master 127.4.49.62:36695 requested a full tablet report, sending...
I20250905 07:54:15.171278 10616 ts_manager.cc:194] Registered new tserver with Master: fbcc842e26d44f7f8dfc24288a3846a5 (127.4.49.1:41059)
I20250905 07:54:15.172605 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:43757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:15.200870 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36695
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:15.310047 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:15.310303 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:15.310362 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:15.313771 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:15.313855 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:15.313941 10820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:15.318778 10820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:15.319880 10820 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:15.320802 10820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:15.327996 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:15.327996 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:15.328001 10825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:15.328258 10820 server_base.cc:1047] running on GCE node
I20250905 07:54:15.328755 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:15.329314 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:15.330472 10820 hybrid_clock.cc:648] HybridClock initialized: now 1757058855330446 us; error 35 us; skew 500 ppm
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.333396 10820 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:54:15.334576 10820 webserver.cc:480] Webserver started at http://127.4.49.2:37903/ using document root <none> and password file <none>
I20250905 07:54:15.335148 10820 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:15.335204 10820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:15.335392 10820 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:15.337247 10820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "df064c58fadd4bfda70a29f3ccb0329f"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "d3a7c08268f49b2ba775515839d98deb"
server_key_iv: "de2500159f4b4d28c389eb6a6969ea88"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.337774 10820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "df064c58fadd4bfda70a29f3ccb0329f"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "d3a7c08268f49b2ba775515839d98deb"
server_key_iv: "de2500159f4b4d28c389eb6a6969ea88"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.341414 10820 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:15.343923 10835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.345101 10820 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:15.345239 10820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "df064c58fadd4bfda70a29f3ccb0329f"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "d3a7c08268f49b2ba775515839d98deb"
server_key_iv: "de2500159f4b4d28c389eb6a6969ea88"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.345350 10820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:15.360670 10820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:15.363718 10820 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:15.363911 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:15.364533 10820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:15.365401 10820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:15.365451 10820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.365511 10820 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:15.365537 10820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.374145 10820 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:38157
I20250905 07:54:15.374169 10950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:38157 every 8 connection(s)
I20250905 07:54:15.375175 10820 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:15.376876 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10820
I20250905 07:54:15.376976 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:15.377332 4292 external_mini_cluster.cc:1468] Setting key f98deaa842deb1018d5f7b7213f3a7c1
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:54:15.388598 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.376997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:48327 (local address 127.4.49.62:36695)
0905 07:54:15.377197 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.377202 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.378009 (+ 807us) server_negotiation.cc:408] Connection header received
0905 07:54:15.378943 (+ 934us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.378946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.378987 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.379043 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.380762 (+ 1719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.381260 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.382341 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.382534 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.384915 (+ 2381us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.384939 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.384942 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.384973 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.386473 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.387045 (+ 572us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.387049 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.387051 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.387109 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.387410 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.387414 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.387416 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.387590 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.387677 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.388331 (+ 654us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.388464 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20250905 07:54:15.389497 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.377336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36695 (local address 127.4.49.2:48327)
0905 07:54:15.377854 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:15.377899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:15.378740 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:15.379213 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:15.379225 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:15.379786 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:15.380586 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.380606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.381389 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.381393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:15.382220 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.382230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.382448 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.383080 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:15.383108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:15.384710 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:15.386620 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.386628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.386652 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.386917 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.387206 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.387209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.387211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.387317 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.387703 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:15.387711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:15.388001 (+ 290us) client_negotiation.cc:769] Sending connection context
0905 07:54:15.388283 (+ 282us) client_negotiation.cc:240] Negotiation successful
0905 07:54:15.388610 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":138,"threads_started":1}
I20250905 07:54:15.390697 10951 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36695
I20250905 07:54:15.390944 10951 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:15.391523 10951 heartbeater.cc:507] Master 127.4.49.62:36695 requested a full tablet report, sending...
I20250905 07:54:15.392681 10616 ts_manager.cc:194] Registered new tserver with Master: df064c58fadd4bfda70a29f3ccb0329f (127.4.49.2:38157)
I20250905 07:54:15.393486 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:48327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:15.422266 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36695
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:15.529104 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:15.529330 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:15.529387 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:15.532721 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:15.532794 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:15.532876 10958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:15.537330 10958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:15.538432 10958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:15.539250 10958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:15.545898 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:15.545898 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:15.546037 10958 server_base.cc:1047] running on GCE node
W20250905 07:54:15.546187 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:15.546567 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:15.547147 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:15.548337 10958 hybrid_clock.cc:648] HybridClock initialized: now 1757058855548313 us; error 35 us; skew 500 ppm
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.551175 10958 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:54:15.552294 10958 webserver.cc:480] Webserver started at http://127.4.49.3:38083/ using document root <none> and password file <none>
I20250905 07:54:15.552851 10958 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:15.552902 10958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:15.553082 10958 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:15.554833 10958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "683bde3c12bd468caeb4193a21c3f3e9"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "6baea93c0ed5d91aa5c9f199ddf3bc52"
server_key_iv: "d571e822dcc464a0b0393ea7f9ff3374"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.555301 10958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "683bde3c12bd468caeb4193a21c3f3e9"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "6baea93c0ed5d91aa5c9f199ddf3bc52"
server_key_iv: "d571e822dcc464a0b0393ea7f9ff3374"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.558954 10958 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250905 07:54:15.561394 10973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.562747 10958 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:54:15.562872 10958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "683bde3c12bd468caeb4193a21c3f3e9"
format_stamp: "Formatted at 2025-09-05 07:54:15 on dist-test-slave-mnfb"
server_key: "6baea93c0ed5d91aa5c9f199ddf3bc52"
server_key_iv: "d571e822dcc464a0b0393ea7f9ff3374"
server_key_version: "encryptionkey@0"
I20250905 07:54:15.562979 10958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:15.576185 10958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:15.579051 10958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:15.579238 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:15.579797 10958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:15.580708 10958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:15.580757 10958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.580811 10958 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:15.580837 10958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:15.589378 10958 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:38273
I20250905 07:54:15.589392 11087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:38273 every 8 connection(s)
I20250905 07:54:15.590358 10958 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:15.598066 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 10958
I20250905 07:54:15.598181 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:15.598461 4292 external_mini_cluster.cc:1468] Setting key 4184831624fff3308fe3dbb3f7d99678
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:15.602648 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.592173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:37277 (local address 127.4.49.62:36695)
0905 07:54:15.592311 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.592315 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.593008 (+ 693us) server_negotiation.cc:408] Connection header received
0905 07:54:15.593865 (+ 857us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.593869 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.593929 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.594009 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.595368 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.595896 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.596950 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.597091 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.599475 (+ 2384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.599490 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.599493 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.599522 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.601042 (+ 1520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.601483 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.601485 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.601486 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.601527 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.601737 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.601739 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.601740 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.601892 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.601959 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.602406 (+ 447us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.602509 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:54:15.603348 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.592421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36695 (local address 127.4.49.3:37277)
0905 07:54:15.592863 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:15.592902 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:15.593660 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:15.594192 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:15.594202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:15.594619 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:15.595209 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.595222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.596036 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.596039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:15.596839 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:15.596849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.597031 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.597586 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:15.597608 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:15.599341 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:15.601151 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.601156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.601166 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.601406 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.601586 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:15.601589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:15.601590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:15.601668 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:15.601960 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:15.601964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:15.602204 (+ 240us) client_negotiation.cc:769] Sending connection context
0905 07:54:15.602380 (+ 176us) client_negotiation.cc:240] Negotiation successful
0905 07:54:15.602609 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20250905 07:54:15.604544 11088 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36695
I20250905 07:54:15.604909 11088 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:15.605501 11088 heartbeater.cc:507] Master 127.4.49.62:36695 requested a full tablet report, sending...
I20250905 07:54:15.606537 10616 ts_manager.cc:194] Registered new tserver with Master: 683bde3c12bd468caeb4193a21c3f3e9 (127.4.49.3:38273)
I20250905 07:54:15.607100 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:37277
I20250905 07:54:15.612670 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/test-admin.keytab.
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.627616 4292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20250905 07:54:15.630151 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:15.628923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.4.49.1:41059)
0905 07:54:15.629252 (+ 329us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.629259 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.629285 (+ 26us) server_negotiation.cc:408] Connection header received
0905 07:54:15.629336 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.629341 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.629509 (+ 168us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.629652 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.630029 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53310: BlockingRecv error: recv got EOF from 127.0.0.1:53310 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":133,"threads_started":1}
W20250905 07:54:15.630343 4292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc' of type FILE
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.631426 4292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.632357 4292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:54:15.641350 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.633302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.4.49.1:41059)
0905 07:54:15.633429 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.633434 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.633510 (+ 76us) server_negotiation.cc:408] Connection header received
0905 07:54:15.633659 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.633663 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.633712 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.633780 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.634449 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.635166 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.635841 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.636135 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.638062 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.638092 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.638106 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.638136 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.639646 (+ 1510us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.640047 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.640115 (+ 68us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.640117 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.640179 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.640405 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.640408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.640410 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.640636 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.640839 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.641038 (+ 199us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.641197 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20250905 07:54:15.645051 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:15.644127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.4.49.1:41059)
0905 07:54:15.644235 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.644238 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.644281 (+ 43us) server_negotiation.cc:408] Connection header received
0905 07:54:15.644401 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.644404 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.644446 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.644511 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.644934 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250905 07:54:15.645156 4292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc' of type FILE
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.646179 4292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.647068 4292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:54:15.655019 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.647955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.4.49.1:41059)
0905 07:54:15.648144 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.648148 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.648164 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:15.648290 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.648293 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.648337 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.648411 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.648973 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.649520 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.650211 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.650379 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.652431 (+ 2052us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.652456 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.652458 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.652489 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.653731 (+ 1242us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.654103 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.654105 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.654107 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.654152 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.654335 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.654337 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.654339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.654511 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.654615 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.654790 (+ 175us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.654888 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20250905 07:54:15.657881 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:15.656910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.4.49.1:41059)
0905 07:54:15.657060 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.657064 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.657078 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:15.657198 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.657202 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.657245 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.657316 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.657775 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53352: BlockingRecv error: recv got EOF from 127.0.0.1:53352 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20250905 07:54:15.657989 4292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1757058797595744-4292-0/krb5kdc/krb5cc' of type FILE
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.659022 4292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:15.659847 4292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.1@KRBTEST.COM
I20250905 07:54:15.667761 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:15.660816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.4.49.1:41059)
0905 07:54:15.660964 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:15.660968 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:15.661012 (+ 44us) server_negotiation.cc:408] Connection header received
0905 07:54:15.661122 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:15.661125 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:15.661169 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:15.661237 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:15.661851 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.662318 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:15.662882 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:15.663006 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:15.665008 (+ 2002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:15.665033 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:15.665035 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:15.665068 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:15.666452 (+ 1384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.666861 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.666863 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.666865 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.666910 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:15.667071 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:15.667073 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:15.667074 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:15.667218 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:15.667329 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:15.667475 (+ 146us) server_negotiation.cc:300] Negotiation successful
0905 07:54:15.667572 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:54:15.668764 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10681
I20250905 07:54:15.673820 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10820
I20250905 07:54:15.678776 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10958
I20250905 07:54:15.684036 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 10584
2025-09-05T07:54:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2982 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[11108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[11108](info): set up 2 sockets
Sep 05 07:54:15 dist-test-slave-mnfb krb5kdc[11108](info): commencing operation
krb5kdc: starting...
W20250905 07:54:17.514902 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.810s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:17 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:17.648011 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43263
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:43263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:17.758895 11124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:17.759163 11124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:17.759228 11124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:17.762683 11124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:17.762773 11124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:17.762795 11124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:17.762814 11124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:17.762831 11124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:17.767458 11124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:43263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:43263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:17.768661 11124 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:17.769557 11124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:17.775203 11132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:17.775244 11129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:17.775296 11124 server_base.cc:1047] running on GCE node
W20250905 07:54:17.775249 11130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:17.776489 11124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:17.777658 11124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:17.778885 11124 hybrid_clock.cc:648] HybridClock initialized: now 1757058857778867 us; error 44 us; skew 500 ppm
Sep 05 07:54:17 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:17.781570 11124 init.cc:377] Logged in from keytab as oryx/127.4.49.62@KRBTEST.COM (short username oryx)
I20250905 07:54:17.782739 11124 webserver.cc:480] Webserver started at http://127.4.49.62:36803/ using document root <none> and password file <none>
I20250905 07:54:17.783351 11124 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:17.783411 11124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:17.783607 11124 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:17.785514 11124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd"
format_stamp: "Formatted at 2025-09-05 07:54:17 on dist-test-slave-mnfb"
server_key: "7beddae2f303e9faff04909c80f67045"
server_key_iv: "e77071a37e81fc6bad027d3decccf4a5"
server_key_version: "encryptionkey@0"
I20250905 07:54:17.786042 11124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd"
format_stamp: "Formatted at 2025-09-05 07:54:17 on dist-test-slave-mnfb"
server_key: "7beddae2f303e9faff04909c80f67045"
server_key_iv: "e77071a37e81fc6bad027d3decccf4a5"
server_key_version: "encryptionkey@0"
I20250905 07:54:17.789736 11124 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:17.792034 11139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:17.793247 11124 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:17.793402 11124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd"
format_stamp: "Formatted at 2025-09-05 07:54:17 on dist-test-slave-mnfb"
server_key: "7beddae2f303e9faff04909c80f67045"
server_key_iv: "e77071a37e81fc6bad027d3decccf4a5"
server_key_version: "encryptionkey@0"
I20250905 07:54:17.793505 11124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:17.817796 11124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:17.820555 11124 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:17.820731 11124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:17.828233 11124 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:43263
I20250905 07:54:17.828233 11192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:43263 every 8 connection(s)
I20250905 07:54:17.829298 11124 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:17.832136 11193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:17.834125 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11124
I20250905 07:54:17.834241 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:17.834453 4292 external_mini_cluster.cc:1468] Setting key 51c7f0c8d929c3d0d52ebab6aadc5a6f
I20250905 07:54:17.838729 11193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Bootstrap starting.
I20250905 07:54:17.841293 11193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:17.842149 11193 log.cc:826] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log is configured to *not* fsync() on all Append() calls
Sep 05 07:54:17 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:17.844437 11193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: No bootstrap required, opened a new log
I20250905 07:54:17.847339 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:17.835734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.4.49.62:43263)
0905 07:54:17.836191 (+ 457us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:17.836204 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:17.836246 (+ 42us) server_negotiation.cc:408] Connection header received
0905 07:54:17.836969 (+ 723us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:17.836999 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:17.837417 (+ 418us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:17.837812 (+ 395us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:17.838564 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:17.839676 (+ 1112us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:17.840366 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:17.840648 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:17.842769 (+ 2121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:17.842787 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:17.842798 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:17.842832 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:17.844703 (+ 1871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:17.845093 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:17.845099 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:17.845104 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:17.845200 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:17.845393 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:17.845396 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:17.845397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:17.845844 (+ 447us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:17.846030 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:17.846273 (+ 243us) server_negotiation.cc:300] Negotiation successful
0905 07:54:17.846548 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20250905 07:54:17.847635 11193 raft_consensus.cc:357] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } }
I20250905 07:54:17.847858 11193 raft_consensus.cc:383] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:17.847918 11193 raft_consensus.cc:738] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5f3ac0b28c4ff9ad224ec8ae03a2cd, State: Initialized, Role: FOLLOWER
I20250905 07:54:17.848394 11193 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } }
I20250905 07:54:17.848555 11193 raft_consensus.cc:397] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:17.848623 11193 raft_consensus.cc:491] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:17.848735 11193 raft_consensus.cc:3058] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:17.849697 11193 raft_consensus.cc:513] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } }
I20250905 07:54:17.850019 11193 leader_election.cc:304] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa5f3ac0b28c4ff9ad224ec8ae03a2cd; no voters:
I20250905 07:54:17.850298 11193 leader_election.cc:290] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:17.850468 11198 raft_consensus.cc:2802] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:17.850719 11198 raft_consensus.cc:695] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 LEADER]: Becoming Leader. State: Replica: aa5f3ac0b28c4ff9ad224ec8ae03a2cd, State: Running, Role: LEADER
I20250905 07:54:17.851013 11198 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } }
I20250905 07:54:17.851374 11193 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:17.852651 11200 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } }
I20250905 07:54:17.852787 11200 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:17.853134 11207 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:17.853818 11199 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } }
I20250905 07:54:17.853947 11199 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:17.856269 11207 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:17.861918 11207 catalog_manager.cc:1349] Generated new cluster ID: 90f4d38706574efd912be68b1e4d1b22
I20250905 07:54:17.862007 11207 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:17.875689 11207 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:17.876770 11207 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:17.883347 11207 catalog_manager.cc:5955] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Generated new TSK 0
I20250905 07:54:17.884156 11207 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:17.940438 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43263
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:18.070116 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:18.070374 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:18.070436 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:18.073807 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:18.073896 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:18.073994 11221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:18.078474 11221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:18.079584 11221 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:18.080503 11221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:18.086894 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.086905 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.086894 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:18.087139 11221 server_base.cc:1047] running on GCE node
I20250905 07:54:18.087544 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:18.088132 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:18.089273 11221 hybrid_clock.cc:648] HybridClock initialized: now 1757058858089260 us; error 31 us; skew 500 ppm
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:18.091830 11221 init.cc:377] Logged in from keytab as oryx/127.4.49.1@KRBTEST.COM (short username oryx)
I20250905 07:54:18.092963 11221 webserver.cc:480] Webserver started at http://127.4.49.1:36719/ using document root <none> and password file <none>
I20250905 07:54:18.093545 11221 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.093614 11221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.093818 11221 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:18.095659 11221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "fba28b8cfc4c466d3857db66ead34f93"
server_key_iv: "f8d5047e6ec8943f9a7c7ceb5c1f52cb"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.096176 11221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "fba28b8cfc4c466d3857db66ead34f93"
server_key_iv: "f8d5047e6ec8943f9a7c7ceb5c1f52cb"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.099637 11221 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:18.101923 11236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.102945 11221 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:18.103060 11221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "fba28b8cfc4c466d3857db66ead34f93"
server_key_iv: "f8d5047e6ec8943f9a7c7ceb5c1f52cb"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.103152 11221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:18.118099 11221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:18.120944 11221 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:18.121124 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:18.121687 11221 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:18.122578 11221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:18.122627 11221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.122679 11221 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:18.122705 11221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.131407 11221 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:35795
I20250905 07:54:18.131445 11350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:35795 every 8 connection(s)
I20250905 07:54:18.132395 11221 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:18.137234 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11221
I20250905 07:54:18.137391 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:18.137655 4292 external_mini_cluster.cc:1468] Setting key d188a1a6d6666c47127df14cc0f965b9
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:18.145407 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.134214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:50651 (local address 127.4.49.62:43263)
0905 07:54:18.134386 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.134392 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.134965 (+ 573us) server_negotiation.cc:408] Connection header received
0905 07:54:18.135795 (+ 830us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.135799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.135852 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.135923 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.137855 (+ 1932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.138436 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.139212 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.139357 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.142008 (+ 2651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.142026 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.142029 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.142057 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.143485 (+ 1428us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.143985 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.143988 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.143990 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.144038 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.144328 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.144332 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.144334 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.144503 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.144588 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.145095 (+ 507us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.145233 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20250905 07:54:18.146100 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.134393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:50651)
0905 07:54:18.134830 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.134867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.135600 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.136102 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.136110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.136518 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.137694 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.137715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.138557 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.138561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.139086 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.139097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.139306 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.139935 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.139970 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.141835 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.143603 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.143609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.143621 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.143881 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.144143 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.144147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.144149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.144243 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.144600 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.144608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.144856 (+ 248us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.145049 (+ 193us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.145337 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:54:18.147455 11351 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:18.147702 11351 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:18.148312 11351 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.2@KRBTEST.COM" created.
I20250905 07:54:18.150020 11156 ts_manager.cc:194] Registered new tserver with Master: 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
I20250905 07:54:18.151239 11156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:50651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:18.186510 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43263
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:18.295802 11358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:18.296105 11358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:18.296168 11358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:18.299712 11358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:18.299806 11358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:18.299909 11358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:18.304632 11358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:18.305887 11358 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:18.306797 11358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:18.313618 11363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.313622 11364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.313618 11366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:18.313882 11358 server_base.cc:1047] running on GCE node
I20250905 07:54:18.314260 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:18.314802 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:18.315949 11358 hybrid_clock.cc:648] HybridClock initialized: now 1757058858315938 us; error 32 us; skew 500 ppm
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:18.318535 11358 init.cc:377] Logged in from keytab as oryx/127.4.49.2@KRBTEST.COM (short username oryx)
I20250905 07:54:18.319620 11358 webserver.cc:480] Webserver started at http://127.4.49.2:37677/ using document root <none> and password file <none>
I20250905 07:54:18.320232 11358 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.320288 11358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.320482 11358 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:18.322377 11358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "477f5d65c7ca4465ac2287c067ffbfdc"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "4b1ed45adb8826299a5e6ab6be4a88c8"
server_key_iv: "d5cc17621eb90173401d36a83f546c8b"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.322886 11358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "477f5d65c7ca4465ac2287c067ffbfdc"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "4b1ed45adb8826299a5e6ab6be4a88c8"
server_key_iv: "d5cc17621eb90173401d36a83f546c8b"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.326668 11358 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250905 07:54:18.329226 11373 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.330333 11358 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:18.330457 11358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "477f5d65c7ca4465ac2287c067ffbfdc"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "4b1ed45adb8826299a5e6ab6be4a88c8"
server_key_iv: "d5cc17621eb90173401d36a83f546c8b"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.330564 11358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:18.355852 11358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:18.359117 11358 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:18.359313 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:18.359897 11358 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:18.360857 11358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:18.360908 11358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.360967 11358 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:18.360991 11358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.369619 11358 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:36415
I20250905 07:54:18.369690 11486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:36415 every 8 connection(s)
I20250905 07:54:18.370635 11358 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:18.372720 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11358
I20250905 07:54:18.372804 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:18.373013 4292 external_mini_cluster.cc:1468] Setting key 6134fe70f1a20c03b074409c9460a2e2
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.3@KRBTEST.COM" created.
I20250905 07:54:18.384724 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.372500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:57425 (local address 127.4.49.62:43263)
0905 07:54:18.372670 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.372675 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.373392 (+ 717us) server_negotiation.cc:408] Connection header received
0905 07:54:18.374290 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.374293 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.374351 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.374419 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.376637 (+ 2218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.377386 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.378069 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.378282 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.380648 (+ 2366us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.380674 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.380678 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.380713 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.382299 (+ 1586us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.382857 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.382860 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.382862 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.382912 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.383277 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.383280 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.383282 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.383547 (+ 265us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.383650 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.384384 (+ 734us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.384548 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:54:18.385449 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.372782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:57425)
0905 07:54:18.373242 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.373289 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.374107 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.374601 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.374613 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.375190 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.376483 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.376502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.377545 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.377549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.377954 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.377961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.378172 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.378833 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.378859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.380467 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.382454 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.382461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.382474 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.382750 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.383020 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.383030 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.383032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.383182 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.383687 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.383702 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.384006 (+ 304us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.384293 (+ 287us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.384603 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
I20250905 07:54:18.386588 11487 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:18.386870 11487 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:18.387450 11487 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
I20250905 07:54:18.388536 11156 ts_manager.cc:194] Registered new tserver with Master: 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
I20250905 07:54:18.389124 11156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:57425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:18.419930 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:43263
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:18.527647 11494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:18.527897 11494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:18.527958 11494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:18.531343 11494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:18.531428 11494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:18.531526 11494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:18.536015 11494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:18.537196 11494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:18.538113 11494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:18.544699 11502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.544701 11499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.544701 11500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:18.544922 11494 server_base.cc:1047] running on GCE node
I20250905 07:54:18.545313 11494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:18.545905 11494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:18.547115 11494 hybrid_clock.cc:648] HybridClock initialized: now 1757058858547094 us; error 33 us; skew 500 ppm
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:18.549943 11494 init.cc:377] Logged in from keytab as oryx/127.4.49.3@KRBTEST.COM (short username oryx)
I20250905 07:54:18.551134 11494 webserver.cc:480] Webserver started at http://127.4.49.3:39135/ using document root <none> and password file <none>
I20250905 07:54:18.551707 11494 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.551757 11494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.551927 11494 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:18.553875 11494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "d83ef27ae9e547a1afe043b7a0b956ea"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "f08ea9c07e66d1a4565b388dc2e68b90"
server_key_iv: "f042d30e7653366dbbcd745802fda199"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.554330 11494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "d83ef27ae9e547a1afe043b7a0b956ea"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "f08ea9c07e66d1a4565b388dc2e68b90"
server_key_iv: "f042d30e7653366dbbcd745802fda199"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.557862 11494 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:18.560192 11509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.561260 11494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:18.561365 11494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "d83ef27ae9e547a1afe043b7a0b956ea"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "f08ea9c07e66d1a4565b388dc2e68b90"
server_key_iv: "f042d30e7653366dbbcd745802fda199"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.561452 11494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:18.582105 11494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:18.585179 11494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:18.585345 11494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:18.585873 11494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:18.586741 11494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:18.586789 11494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.586830 11494 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:18.586850 11494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.595674 11494 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:37791
I20250905 07:54:18.595754 11624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:37791 every 8 connection(s)
I20250905 07:54:18.596691 11494 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:18.605821 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11494
I20250905 07:54:18.605906 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:18.606384 4292 external_mini_cluster.cc:1468] Setting key daa483ea544cfb8e7c7112a7e8cca1ba
I20250905 07:54:18.609126 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.598479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:46381 (local address 127.4.49.62:43263)
0905 07:54:18.598605 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.598609 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.599320 (+ 711us) server_negotiation.cc:408] Connection header received
0905 07:54:18.600219 (+ 899us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.600222 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.600288 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.600357 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.602090 (+ 1733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.602676 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.603326 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.603513 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.605672 (+ 2159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.605695 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.605697 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.605749 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.607114 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.607696 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.607699 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.607700 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.607750 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.608022 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.608026 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.608027 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.608231 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.608320 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.608832 (+ 512us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.608961 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250905 07:54:18.609757 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.598754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:46381)
0905 07:54:18.599170 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.599209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.599991 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.600543 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.600552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.600969 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.601925 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.601942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.602801 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.602804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.603204 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.603211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.603428 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.604030 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.604089 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.605512 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.607240 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.607246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.607260 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.607597 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.607833 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.607836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.607839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.607939 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.608332 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.608339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.608584 (+ 245us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.608783 (+ 199us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.609066 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20250905 07:54:18.610806 11625 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:18.611052 11625 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:18.611598 11625 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
I20250905 07:54:18.612627 11156 ts_manager.cc:194] Registered new tserver with Master: d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791)
I20250905 07:54:18.613175 11156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:46381
I20250905 07:54:18.619980 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:18.662925 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.61:33881
--webserver_interface=127.4.49.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:43263,127.4.49.61:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:18.771430 11632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:18.771695 11632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:18.771755 11632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:18.775087 11632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:18.775173 11632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:18.775204 11632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:18.775228 11632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:18.775260 11632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:18.780170 11632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:43263,127.4.49.61:33881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.61:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.49.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
W20250905 07:54:18.780483 11632 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.49.62:43263,127.4.49.61:33881'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20250905 07:54:18.781718 11632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:18.782603 11632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:18.788247 11640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.788270 11638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.788268 11637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:18.788491 11632 server_base.cc:1047] running on GCE node
I20250905 07:54:18.788986 11632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:18.789899 11632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:18.791062 11632 hybrid_clock.cc:648] HybridClock initialized: now 1757058858791045 us; error 35 us; skew 500 ppm
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:18.793743 11632 init.cc:377] Logged in from keytab as oryx/127.4.49.61@KRBTEST.COM (short username oryx)
I20250905 07:54:18.794829 11632 webserver.cc:480] Webserver started at http://127.4.49.61:46883/ using document root <none> and password file <none>
I20250905 07:54:18.795382 11632 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.795432 11632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.795603 11632 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:18.797400 11632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/instance:
uuid: "0ba70538fb1d4086953b29acbf711580"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "8120caff6f902a83c9aadfae56e178a8"
server_key_iv: "dc4be6fc9901d7d19179e12e69f30e59"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.797943 11632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal/instance:
uuid: "0ba70538fb1d4086953b29acbf711580"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "8120caff6f902a83c9aadfae56e178a8"
server_key_iv: "dc4be6fc9901d7d19179e12e69f30e59"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.801235 11632 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:18.803387 11647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.804518 11632 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:18.804630 11632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
uuid: "0ba70538fb1d4086953b29acbf711580"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "8120caff6f902a83c9aadfae56e178a8"
server_key_iv: "dc4be6fc9901d7d19179e12e69f30e59"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.804731 11632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:18.824441 11632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:18.827528 11632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:18.827739 11632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:18.835683 11632 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.61:33881
I20250905 07:54:18.835683 11700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.61:33881 every 8 connection(s)
I20250905 07:54:18.836812 11632 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/info.pb
I20250905 07:54:18.838521 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11632
I20250905 07:54:18.840042 11701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:18.843513 11701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } has no permanent_uuid. Determining permanent_uuid...
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.61@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:18.852519 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.839917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.4.49.61:33881)
0905 07:54:18.840483 (+ 566us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.840494 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.840532 (+ 38us) server_negotiation.cc:408] Connection header received
0905 07:54:18.841612 (+ 1080us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.841634 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.841950 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.842278 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.843224 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.844033 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.844718 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.844991 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.847422 (+ 2431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.847503 (+ 81us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.847514 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.847559 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.849705 (+ 2146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.850176 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.850181 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.850186 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.850252 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.850492 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.850495 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.850497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.850930 (+ 433us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.851131 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.851419 (+ 288us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.851671 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":164,"threads_started":1}
I20250905 07:54:18.853639 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.845145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.4.49.62:43263)
0905 07:54:18.845291 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.845296 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.845622 (+ 326us) server_negotiation.cc:408] Connection header received
0905 07:54:18.845749 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.845753 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.845815 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.845909 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.846835 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.847342 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.848161 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.848322 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.850405 (+ 2083us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.850425 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.850427 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.850457 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.851902 (+ 1445us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.852438 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.852442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.852443 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.852486 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.852760 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.852764 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.852766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.852975 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.853059 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.853365 (+ 306us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.853485 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:54:18.853649 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.845089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39962)
0905 07:54:18.845493 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.845538 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.845658 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.845959 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.845963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.846363 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.846703 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.846712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.847465 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.847468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.848002 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.848011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.848140 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.848607 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.848622 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.850279 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.852005 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.852011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.852015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.852345 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.852569 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.852571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.852574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.852631 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.853074 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.853080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.853193 (+ 113us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.853332 (+ 139us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.853483 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":149,"threads_started":1}
I20250905 07:54:18.854841 11701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } has no permanent_uuid. Determining permanent_uuid...
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.61@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
I20250905 07:54:18.861351 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.855296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51692)
0905 07:54:18.855414 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.855429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.855528 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.855723 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.855726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.855885 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.856072 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.856080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.856700 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.856703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.857034 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.857040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.857133 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.857609 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.857621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.858713 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.860164 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.860167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.860170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.860321 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.860507 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.860511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.860513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.860558 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.860861 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.860864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.860936 (+ 72us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.861021 (+ 85us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.861162 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250905 07:54:18.861351 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.855343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.4.49.61:33881)
0905 07:54:18.855425 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.855429 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.855494 (+ 65us) server_negotiation.cc:408] Connection header received
0905 07:54:18.855605 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.855608 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.855640 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.855704 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.856175 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.856612 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.857123 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.857285 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.858817 (+ 1532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.858831 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.858834 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.858853 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.860073 (+ 1220us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.860397 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.860401 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.860403 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.860438 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.860624 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.860627 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.860629 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.860782 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.860863 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.861066 (+ 203us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.861217 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":30}
I20250905 07:54:18.864279 11701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Bootstrap starting.
I20250905 07:54:18.866451 11701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:18.867425 11701 log.cc:826] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:18.869278 11701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: No bootstrap required, opened a new log
I20250905 07:54:18.871742 11701 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } }
I20250905 07:54:18.871922 11701 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:18.871969 11701 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba70538fb1d4086953b29acbf711580, State: Initialized, Role: FOLLOWER
I20250905 07:54:18.872386 11701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } }
I20250905 07:54:18.872817 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } }
I20250905 07:54:18.872941 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:18.873446 11701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: configured and running, proceeding with master startup.
W20250905 07:54:18.878070 11718 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:54:18.878171 11718 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:54:18.879559 11632 master_runner.cc:428] Detected that this master 0ba70538fb1d4086953b29acbf711580 is joining an existing cluster
I20250905 07:54:18.879617 11632 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.49.61:33881
I20250905 07:54:18.889158 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.881500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39964)
0905 07:54:18.881900 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.881915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.882025 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.882271 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.882274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.882529 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.883034 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.883043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.884098 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.884102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.884873 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.884883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.884991 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.885479 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.885488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.885706 (+ 218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.887722 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.887725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.887726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.887890 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.888159 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.888161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.888163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.888206 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.888588 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.888591 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.888676 (+ 85us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.888873 (+ 197us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.888997 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":114,"threads_started":1}
I20250905 07:54:18.889237 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.881568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.4.49.62:43263)
0905 07:54:18.881676 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.881680 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.881998 (+ 318us) server_negotiation.cc:408] Connection header received
0905 07:54:18.882128 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.882131 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.882183 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.882247 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.883161 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.883947 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.885012 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.885223 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.885881 (+ 658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.885904 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.885908 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.885941 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.887596 (+ 1655us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.887981 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.887985 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.887987 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.888043 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.888282 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.888285 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.888288 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.888496 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.888599 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.888943 (+ 344us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.889070 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.62@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
I20250905 07:54:18.911877 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.881731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51698)
0905 07:54:18.881936 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.881947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.882054 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.882379 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.882382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.882625 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.882953 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.882963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.883715 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.883717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.884140 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.884148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.884247 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.884794 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.884811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.885199 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.910292 (+ 25093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.910295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.910297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.910628 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.910856 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.910858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.910860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.910903 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.911336 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.911339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.911420 (+ 81us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.911527 (+ 107us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.911665 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20250905 07:54:18.912768 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.881547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.4.49.61:33881)
0905 07:54:18.881660 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.881665 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.882001 (+ 336us) server_negotiation.cc:408] Connection header received
0905 07:54:18.882165 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.882168 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.882208 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.882272 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.883078 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.883594 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.884245 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.884439 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.885294 (+ 855us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.885309 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.885312 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.885331 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.910129 (+ 24798us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.910733 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.910736 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.910738 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.910780 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.910982 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.910985 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.910986 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.911211 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.911303 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.912536 (+ 1233us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.912646 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250905 07:54:18.912762 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.893766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51702)
0905 07:54:18.894276 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.894320 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.894485 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.894797 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.894804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.895333 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.895647 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.895658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.896712 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.896716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.897291 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.897301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.897439 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.898031 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.898049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.899757 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.910884 (+ 11127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.910900 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.910907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.911298 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.911568 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.911572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.911574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.911635 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.912078 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.912086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.912237 (+ 151us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.912393 (+ 156us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.912582 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":77,"threads_started":1}
I20250905 07:54:18.913443 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.893830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.4.49.61:33881)
0905 07:54:18.894148 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.894153 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.894392 (+ 239us) server_negotiation.cc:408] Connection header received
0905 07:54:18.894605 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.894607 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.894656 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.894738 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.895766 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.896606 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.897410 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.897582 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.899915 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.899938 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.899941 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.899972 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.910749 (+ 10777us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.911401 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.911404 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.911406 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.911473 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.911745 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.911749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.911751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.911902 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.911984 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.913186 (+ 1202us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.913296 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":29056,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for oryx/127.4.49.60@KRBTEST.COM; defaulting to no policy
I20250905 07:54:18.914633 11156 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.4.49.61:33881: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ba70538fb1d4086953b29acbf711580"
member_type: NON_VOTER
last_known_addr {
host: "127.4.49.61"
port: 33881
}
attrs {
promote: true
}
}
dest_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd"
cas_config_opid_index: -1
I20250905 07:54:18.915234 11156 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.60@KRBTEST.COM" created.
I20250905 07:54:18.917106 11732 raft_consensus.cc:2953] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0ba70538fb1d4086953b29acbf711580 (127.4.49.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:18.918828 11675 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:18.920311 11675 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250905 07:54:18.921025 11731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:18.921185 11732 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.4.49.61:33881
I20250905 07:54:18.921680 11632 master.cc:561] Master@127.4.49.61:33881 shutting down...
W20250905 07:54:18.922825 11143 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20250905 07:54:18.923868 11143 consensus_peers.cc:489] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd -> Peer 0ba70538fb1d4086953b29acbf711580 (127.4.49.61:33881): Couldn't send request to peer 0ba70538fb1d4086953b29acbf711580. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:54:18.925992 11632 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:18.925983 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:18.926115 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:18.926280 11632 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:18.926394 11632 tablet_replica.cc:331] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:18.940954 11632 master.cc:583] Master@127.4.49.61:33881 shutdown complete.
I20250905 07:54:18.941020 11632 master_runner.cc:305] Clearing existing system tablet
I20250905 07:54:18.941228 11632 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.941280 11632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.942389 11632 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
WARNING: no policy specified for HTTP/127.4.49.60@KRBTEST.COM; defaulting to no policy
I20250905 07:54:18.943186 11739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.943369 11632 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:18.943495 11632 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:18.943529 11632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
uuid: "0ba70538fb1d4086953b29acbf711580"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "8120caff6f902a83c9aadfae56e178a8"
server_key_iv: "dc4be6fc9901d7d19179e12e69f30e59"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.60@KRBTEST.COM" created.
I20250905 07:54:18.946015 11632 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:18.948115 11632 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250905 07:54:18.948212 11632 log.cc:1199] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20250905 07:54:18.948427 11632 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Deleting consensus metadata
I20250905 07:54:18.948586 11632 master_runner.cc:315] Copying system tablet from 127.4.49.62:43263
I20250905 07:54:18.949697 11632 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: tablet copy: Beginning tablet copy session from remote peer at address 127.4.49.62:43263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:18.960397 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.950300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.4.49.62:43263)
0905 07:54:18.950440 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:18.950444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:18.950637 (+ 193us) server_negotiation.cc:408] Connection header received
0905 07:54:18.950768 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:18.950772 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:18.950822 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:18.950887 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:18.951444 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.951988 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.952662 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.952865 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.953705 (+ 840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:18.953725 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:18.953728 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:18.953756 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:18.958572 (+ 4816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.959127 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.959130 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.959131 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.959184 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:18.959419 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:18.959422 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:18.959424 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:18.959619 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:18.959708 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:18.960044 (+ 336us) server_negotiation.cc:300] Negotiation successful
0905 07:54:18.960213 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:54:18.960850 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.60:45847
--webserver_interface=127.4.49.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250905 07:54:18.961007 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:18.950210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39968)
0905 07:54:18.950553 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:18.950564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:18.950677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:18.950925 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:18.950928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:18.951113 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:18.951322 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.951328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.952117 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:18.952121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:18.952546 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:18.952554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:18.952657 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:18.953278 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:18.953292 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:18.953578 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:18.958746 (+ 5168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.958751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.958753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.959000 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.959261 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:18.959264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:18.959266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:18.959323 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:18.959716 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:18.959719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:18.959826 (+ 107us) client_negotiation.cc:769] Sending connection context
0905 07:54:18.960657 (+ 831us) client_negotiation.cc:240] Negotiation successful
0905 07:54:18.960823 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":94,"threads_started":1}
I20250905 07:54:18.962770 11177 tablet_copy_service.cc:140] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ba70538fb1d4086953b29acbf711580 ({username='oryx', principal='oryx/127.4.49.61@KRBTEST.COM'} at 127.0.0.1:39968)
I20250905 07:54:18.962944 11177 tablet_copy_service.cc:161] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ba70538fb1d4086953b29acbf711580 at {username='oryx', principal='oryx/127.4.49.61@KRBTEST.COM'} at 127.0.0.1:39968: session id = 0ba70538fb1d4086953b29acbf711580-00000000000000000000000000000000
I20250905 07:54:18.964736 11177 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 07:54:18.966270 11632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250905 07:54:18.968019 11632 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: tablet copy: Starting download of 0 data blocks...
I20250905 07:54:18.968235 11632 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: tablet copy: Starting download of 1 WAL segments...
I20250905 07:54:18.969990 11632 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 07:54:18.971562 11177 tablet_copy_service.cc:342] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Request end of tablet copy session 0ba70538fb1d4086953b29acbf711580-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.49.61@KRBTEST.COM'} at 127.0.0.1:39968
I20250905 07:54:18.971683 11177 tablet_copy_service.cc:434] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: ending tablet copy session 0ba70538fb1d4086953b29acbf711580-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ba70538fb1d4086953b29acbf711580
W20250905 07:54:18.976725 11632 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 07:54:18.978708 11632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:18.980201 11751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.980306 11748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:18.980373 11749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:18.980580 11632 server_base.cc:1047] running on GCE node
I20250905 07:54:18.980720 11632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:18.980969 11632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:18.982097 11632 hybrid_clock.cc:648] HybridClock initialized: now 1757058858982090 us; error 24 us; skew 500 ppm
Sep 05 07:54:18 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:18.983497 11632 init.cc:377] Logged in from keytab as oryx/127.4.49.61@KRBTEST.COM (short username oryx)
I20250905 07:54:18.983970 11632 webserver.cc:480] Webserver started at http://127.4.49.61:35425/ using document root <none> and password file <none>
I20250905 07:54:18.984238 11632 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:18.984277 11632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:18.985380 11632 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:18.986181 11758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:18.986400 11632 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:18.986464 11632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
uuid: "0ba70538fb1d4086953b29acbf711580"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "8120caff6f902a83c9aadfae56e178a8"
server_key_iv: "dc4be6fc9901d7d19179e12e69f30e59"
server_key_version: "encryptionkey@0"
I20250905 07:54:18.986543 11632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:19.006350 11632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:19.008322 11632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:19.012727 11632 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.61:33881
I20250905 07:54:19.013257 11632 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-1/data/info.pb
I20250905 07:54:19.013625 11810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.61:33881 every 8 connection(s)
I20250905 07:54:19.014715 11811 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:54:19.015537 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Bootstrap starting.
I20250905 07:54:19.023448 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:54:19.023736 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: Bootstrap complete.
I20250905 07:54:19.024326 11811 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } }
I20250905 07:54:19.024447 11811 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ba70538fb1d4086953b29acbf711580, State: Initialized, Role: LEARNER
I20250905 07:54:19.024555 11811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } }
I20250905 07:54:19.024876 11814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:19.025024 11814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: LEARNER
I20250905 07:54:19.025023 11811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:19.028600 11825 catalog_manager.cc:1261] Loaded cluster ID: 90f4d38706574efd912be68b1e4d1b22
I20250905 07:54:19.028656 11825 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: loading cluster ID for follower catalog manager: success
I20250905 07:54:19.030243 11825 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: acquiring CA information for follower catalog manager: success
I20250905 07:54:19.031366 11825 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250905 07:54:19.102075 11747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:19.102325 11747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:19.102383 11747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:19.105798 11747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:19.105873 11747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:19.105897 11747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:19.105918 11747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:19.105952 11747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:19.110517 11747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.60:45847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.49.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:19.111646 11747 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:19.112576 11747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:19.117980 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:19.118067 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:19.118145 11747 server_base.cc:1047] running on GCE node
W20250905 07:54:19.118067 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:19.118783 11747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:19.119655 11747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:19.120790 11747 hybrid_clock.cc:648] HybridClock initialized: now 1757058859120775 us; error 34 us; skew 500 ppm
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:19.123378 11747 init.cc:377] Logged in from keytab as oryx/127.4.49.60@KRBTEST.COM (short username oryx)
I20250905 07:54:19.124480 11747 webserver.cc:480] Webserver started at http://127.4.49.60:33749/ using document root <none> and password file <none>
I20250905 07:54:19.125063 11747 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:19.125113 11747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:19.125268 11747 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:19.127090 11747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/instance:
uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
format_stamp: "Formatted at 2025-09-05 07:54:19 on dist-test-slave-mnfb"
server_key: "ec334d1334f813a17f876a02d88d7764"
server_key_iv: "d4d7fe5677f16ff59847086ae984dd08"
server_key_version: "encryptionkey@0"
I20250905 07:54:19.127542 11747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal/instance:
uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
format_stamp: "Formatted at 2025-09-05 07:54:19 on dist-test-slave-mnfb"
server_key: "ec334d1334f813a17f876a02d88d7764"
server_key_iv: "d4d7fe5677f16ff59847086ae984dd08"
server_key_version: "encryptionkey@0"
I20250905 07:54:19.131026 11747 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:54:19.133699 11840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:19.134949 11747 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:54:19.135071 11747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
format_stamp: "Formatted at 2025-09-05 07:54:19 on dist-test-slave-mnfb"
server_key: "ec334d1334f813a17f876a02d88d7764"
server_key_iv: "d4d7fe5677f16ff59847086ae984dd08"
server_key_version: "encryptionkey@0"
I20250905 07:54:19.135166 11747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:19.153825 11351 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:19.171200 11747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:19.174124 11747 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:19.174295 11747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:19.181258 11893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.60:45847 every 8 connection(s)
I20250905 07:54:19.181260 11747 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.60:45847
I20250905 07:54:19.182312 11747 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/info.pb
I20250905 07:54:19.185125 11894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:19.187798 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 11747
I20250905 07:54:19.189252 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } has no permanent_uuid. Determining permanent_uuid...
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.60@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:19.200088 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.191082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.4.49.62:43263)
0905 07:54:19.191218 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.191223 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.191515 (+ 292us) server_negotiation.cc:408] Connection header received
0905 07:54:19.191631 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.191634 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.191685 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.191748 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.192830 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.193595 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.194281 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.194494 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.196663 (+ 2169us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.196683 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.196685 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.196716 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.198353 (+ 1637us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.198817 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.198820 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.198822 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.198875 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.199075 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.199078 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.199079 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.199240 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.199335 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.199771 (+ 436us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.199888 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250905 07:54:19.200690 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.189086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.4.49.60:45847)
0905 07:54:19.189487 (+ 401us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.189497 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.189533 (+ 36us) server_negotiation.cc:408] Connection header received
0905 07:54:19.190090 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.190109 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.190403 (+ 294us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.190724 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.191640 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.192765 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.193585 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.193869 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.196136 (+ 2267us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.196159 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.196172 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.196198 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.198062 (+ 1864us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.198492 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.198497 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.198498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.198565 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.198805 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.198808 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.198810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.199245 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.199416 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.199638 (+ 222us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.199909 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20250905 07:54:19.200695 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.190856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39978)
0905 07:54:19.191386 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.191423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.191541 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.191816 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.191823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.192317 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.192656 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.192669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.193696 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.193700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.194168 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.194176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.194288 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.194730 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.194752 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.196523 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.198460 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.198469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.198478 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.198711 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.198958 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.198960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.198961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.199005 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.199341 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.199347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.199533 (+ 186us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.199705 (+ 172us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.200007 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":98,"thread_start_us":124,"threads_started":1}
I20250905 07:54:19.201942 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } has no permanent_uuid. Determining permanent_uuid...
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.60@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
I20250905 07:54:19.209260 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.202414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51708)
0905 07:54:19.202576 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.202587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.202646 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.203015 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.203019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.203173 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.203352 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.203358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.204325 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.204328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.204731 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.204739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.204852 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.205350 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.205362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.206483 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.208185 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.208188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.208189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.208349 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.208533 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.208535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.208536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.208575 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.208856 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.208858 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.208935 (+ 77us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.209009 (+ 74us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.209115 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20250905 07:54:19.209283 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.202548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.4.49.61:33881)
0905 07:54:19.202820 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.202824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.202834 (+ 10us) server_negotiation.cc:408] Connection header received
0905 07:54:19.202866 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.202869 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.202911 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.202996 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.203488 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.204227 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.204849 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.204985 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.206608 (+ 1623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.206627 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.206629 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.206657 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.208085 (+ 1428us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.208428 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.208430 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.208431 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.208475 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.208630 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.208632 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.208633 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.208794 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.208864 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.209048 (+ 184us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.209152 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20250905 07:54:19.209982 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } has no permanent_uuid. Determining permanent_uuid...
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.60@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
I20250905 07:54:19.216377 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.210449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:53968)
0905 07:54:19.210569 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.210582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.210653 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.210849 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.210851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.210978 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.211143 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.211148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.211744 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.211746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.212113 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.212120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.212201 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.212653 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.212664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.213758 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.215362 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.215364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.215366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.215497 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.215649 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.215651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.215653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.215691 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.215927 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.215929 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.215996 (+ 67us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.216122 (+ 126us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.216230 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250905 07:54:19.216403 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.210578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.4.49.60:45847)
0905 07:54:19.210682 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.210685 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.210695 (+ 10us) server_negotiation.cc:408] Connection header received
0905 07:54:19.210731 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.210734 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.210769 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.210831 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.211219 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.211666 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.212186 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.212322 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.213870 (+ 1548us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.213887 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.213889 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.213910 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.215283 (+ 1373us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.215559 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.215562 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.215563 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.215597 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.215737 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.215739 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.215740 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.215873 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.215934 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.216159 (+ 225us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.216283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":38}
I20250905 07:54:19.219118 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Bootstrap starting.
I20250905 07:54:19.221606 11894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:19.222335 11894 log.cc:826] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:19.224215 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: No bootstrap required, opened a new log
I20250905 07:54:19.226745 11894 raft_consensus.cc:357] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } }
I20250905 07:54:19.226961 11894 raft_consensus.cc:383] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:19.227005 11894 raft_consensus.cc:738] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ad02d5fd364fd6a0e80253d6953ed0, State: Initialized, Role: FOLLOWER
I20250905 07:54:19.227402 11894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } }
I20250905 07:54:19.227849 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } } }
I20250905 07:54:19.227967 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:19.228538 11894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: configured and running, proceeding with master startup.
W20250905 07:54:19.233207 11911 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:54:19.233307 11911 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:54:19.236516 11747 master_runner.cc:428] Detected that this master 51ad02d5fd364fd6a0e80253d6953ed0 is joining an existing cluster
I20250905 07:54:19.236586 11747 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.49.60:45847
I20250905 07:54:19.259989 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.238995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51714)
0905 07:54:19.239631 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.239644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.239827 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.240216 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.240219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.252424 (+ 12205us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.252797 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.252811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.254129 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.254132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.254673 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.254680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.255119 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.255737 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.255753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.256245 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.258363 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.258367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.258369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.258737 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.259009 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.259012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.259014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.259066 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.259449 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.259453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.259540 (+ 87us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.259634 (+ 94us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.259783 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"spinlock_wait_cycles":14208,"thread_start_us":96,"threads_started":1}
I20250905 07:54:19.260527 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.238857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.4.49.61:33881)
0905 07:54:19.239010 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.239015 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.239720 (+ 705us) server_negotiation.cc:408] Connection header received
0905 07:54:19.239941 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.239945 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.239999 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.240076 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.253039 (+ 12963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.253808 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.254802 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.254991 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.256381 (+ 1390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.256404 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.256407 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.256439 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.258249 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.258855 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.258859 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.258861 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.258916 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.259149 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.259152 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.259155 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.259331 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.259410 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.260238 (+ 828us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.260373 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250905 07:54:19.261233 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.256278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39996)
0905 07:54:19.256529 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.256543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.256620 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.258603 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.258606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.258625 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:19.258879 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.258887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.259935 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.259938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.260815 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.260824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.260907 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.260991 (+ 84us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.261035 (+ 44us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.261109 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20250905 07:54:19.261561 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.239363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.4.49.60:45847)
0905 07:54:19.239481 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.239485 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.239837 (+ 352us) server_negotiation.cc:408] Connection header received
0905 07:54:19.239961 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.239964 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.239998 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.240039 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.253001 (+ 12962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.253773 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.254787 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.254963 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.257505 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.257522 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.257525 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.257551 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.259345 (+ 1794us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.260553 (+ 1208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.260557 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.260559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.260598 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.260783 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.260786 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.260788 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.260938 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.261014 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.261280 (+ 266us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.261392 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:54:19.262138 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.239325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:53980)
0905 07:54:19.239774 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.239785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.239884 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.240191 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.240194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.252424 (+ 12230us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.252797 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.252811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.253951 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.253954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.254653 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.254665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.255384 (+ 719us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.256984 (+ 1600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.257001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.257343 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.260236 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.260240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.260242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.260452 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.260675 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.260677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.260679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.260719 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.261058 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.261061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.261132 (+ 71us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.261820 (+ 688us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.261982 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"spinlock_wait_cycles":3456,"thread_start_us":88,"threads_started":1}
I20250905 07:54:19.263600 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.256734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.4.49.62:43263)
0905 07:54:19.257405 (+ 671us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.257409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.257422 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:19.258430 (+ 1008us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.258434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.258489 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.258980 (+ 491us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:19.259064 (+ 84us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.259827 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.262052 (+ 2225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.262555 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.262633 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.262682 (+ 49us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.262763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":119,"threads_started":1}
I20250905 07:54:19.265878 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.239587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.4.49.62:43263)
0905 07:54:19.239713 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.239718 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.239732 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:19.239779 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.239782 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.239837 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.239909 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.253489 (+ 13580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.254330 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.256505 (+ 2175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.257041 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.262532 (+ 5491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.262551 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.262555 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.262585 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.264422 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.264844 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.264848 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.264850 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.264905 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.265137 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.265140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.265142 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.265295 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.265379 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.265636 (+ 257us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.265750 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250905 07:54:19.265909 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.238748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:39990)
0905 07:54:19.239174 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.239187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.239280 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.240039 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.240042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.252352 (+ 12310us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.253274 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.253289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.255164 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.255168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.256360 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.256372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.257163 (+ 791us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.258103 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.258120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.262388 (+ 4268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.264528 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.264532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.264534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.264725 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.265010 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.265012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.265014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.265058 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.265380 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.265383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.265486 (+ 103us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.265571 (+ 85us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.265715 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20250905 07:54:19.268280 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.264796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51718)
0905 07:54:19.264972 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.264985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.265055 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.265522 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.265525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.265545 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:19.265798 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.265805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.266926 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.266930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.267879 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.267890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.267997 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.268015 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.268090 (+ 75us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.268156 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20250905 07:54:19.268836 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.264893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.4.49.61:33881)
0905 07:54:19.265214 (+ 321us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.265218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.265231 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:19.265274 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.265277 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.265337 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.265418 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:19.266003 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.266792 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.268027 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.268453 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.268608 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.268656 (+ 48us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.268718 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.62@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
I20250905 07:54:19.278246 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.270356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.4.49.60:45847)
0905 07:54:19.270473 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.270477 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.270489 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:54:19.270571 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.270575 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.270624 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.270686 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.271271 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.271778 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.272620 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.272802 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.274790 (+ 1988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.274810 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.274814 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.274845 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.276520 (+ 1675us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.277006 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.277010 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.277012 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.277063 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.277311 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.277315 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.277317 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.277531 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.277603 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.277954 (+ 351us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.278080 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250905 07:54:19.278829 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.270101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:53986)
0905 07:54:19.270230 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.270244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.270320 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.270723 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.270726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.270922 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.271154 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.271162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.271890 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.271893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.272459 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.272469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.272617 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.273260 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.273278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.274673 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.276653 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.276657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.276660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.276894 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.277163 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.277166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.277169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.277224 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.277673 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.277677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.277782 (+ 105us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.278512 (+ 730us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.278664 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250905 07:54:19.279796 11155 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.4.49.60:45847: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
member_type: NON_VOTER
last_known_addr {
host: "127.4.49.60"
port: 45847
}
attrs {
promote: true
}
}
dest_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd"
cas_config_opid_index: 5
I20250905 07:54:19.280369 11155 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
I20250905 07:54:19.281327 11734 raft_consensus.cc:2953] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.282475 11734 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.4.49.60:45847
I20250905 07:54:19.282655 11732 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.282761 11732 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:19.282860 11785 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 LEARNER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250905 07:54:19.283349 11732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 07:54:19.283705 11868 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:19.285652 11868 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250905 07:54:19.285392 11785 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.286213 11732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:19.287657 11814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.287753 11814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: LEARNER
I20250905 07:54:19.287845 11814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.287904 11814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: LEARNER
I20250905 07:54:19.288194 11747 master.cc:561] Master@127.4.49.60:45847 shutting down...
I20250905 07:54:19.290200 11732 raft_consensus.cc:1062] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: attempting to promote NON_VOTER 0ba70538fb1d4086953b29acbf711580 to VOTER
I20250905 07:54:19.290632 11732 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
I20250905 07:54:19.291467 11868 raft_consensus.cc:870] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
W20250905 07:54:19.291564 11868 raft_consensus.cc:1577] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
I20250905 07:54:19.291945 11785 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 LEARNER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250905 07:54:19.292486 11732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250905 07:54:19.292937 11140 consensus_peers.cc:489] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd -> Peer 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60:45847): Couldn't send request to peer 51ad02d5fd364fd6a0e80253d6953ed0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:54:19.293102 11868 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0ba70538fb1d4086953b29acbf711580 (127.4.49.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:19.294401 11731 raft_consensus.cc:2953] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ba70538fb1d4086953b29acbf711580 (127.4.49.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.294886 11926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:19.295045 11926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250905 07:54:19.295161 11926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:19.295272 11926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250905 07:54:19.296154 11732 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.296257 11732 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:19.296813 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: NON_VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: true } } }
I20250905 07:54:19.296901 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250905 07:54:19.295897 11731 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.297201 11731 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:19.297606 11785 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ba70538fb1d4086953b29acbf711580 (127.4.49.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.299582 11922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.299696 11922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:19.300863 11747 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20250905 07:54:19.301121 11747 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
W20250905 07:54:19.301170 11155 master.cc:679] Remote error: unable to get registration information for peer 51ad02d5fd364fd6a0e80253d6953ed0: Service unavailable: service kudu.master.MasterService not registered on Master
I20250905 07:54:19.301199 11747 tablet_replica.cc:331] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: stopping tablet replica
I20250905 07:54:19.314086 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.313738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:53998)
0905 07:54:19.313892 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.313979 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20250905 07:54:19.314314 11155 master.cc:679] Network error: unable to get registration information for peer 51ad02d5fd364fd6a0e80253d6953ed0: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
I20250905 07:54:19.319221 11747 master.cc:583] Master@127.4.49.60:45847 shutdown complete.
I20250905 07:54:19.319283 11747 master_runner.cc:305] Clearing existing system tablet
I20250905 07:54:19.319482 11747 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:19.319532 11747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:19.320745 11747 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250905 07:54:19.321556 11931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:19.321851 11747 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:19.321995 11747 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:19.322037 11747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
format_stamp: "Formatted at 2025-09-05 07:54:19 on dist-test-slave-mnfb"
server_key: "ec334d1334f813a17f876a02d88d7764"
server_key_iv: "d4d7fe5677f16ff59847086ae984dd08"
server_key_version: "encryptionkey@0"
I20250905 07:54:19.323889 11747 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:19.326434 11747 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250905 07:54:19.326542 11747 log.cc:1199] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20250905 07:54:19.326848 11747 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Deleting consensus metadata
I20250905 07:54:19.327066 11747 master_runner.cc:315] Copying system tablet from 127.4.49.62:43263
I20250905 07:54:19.328394 11747 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: tablet copy: Beginning tablet copy session from remote peer at address 127.4.49.62:43263
I20250905 07:54:19.334473 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.334170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:54004)
0905 07:54:19.334310 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.334369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20250905 07:54:19.334712 11155 master.cc:679] Network error: unable to get registration information for peer 51ad02d5fd364fd6a0e80253d6953ed0: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
I20250905 07:54:19.336107 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.329077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.4.49.62:43263)
0905 07:54:19.329229 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.329233 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.329315 (+ 82us) server_negotiation.cc:408] Connection header received
0905 07:54:19.329434 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.329438 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.329519 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.329618 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:19.330221 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.331046 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.331788 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.331970 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.332999 (+ 1029us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:19.333016 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:19.333019 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:19.333046 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:19.334595 (+ 1549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.335011 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.335015 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.335017 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.335091 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:19.335298 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:19.335302 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:19.335304 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:19.335483 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:19.335567 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.335808 (+ 241us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.335907 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20250905 07:54:19.336114 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.328975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40002)
0905 07:54:19.329228 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.329241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.329348 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.329633 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.329635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.329822 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:19.330048 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.330056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.331180 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.331184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.331672 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.331684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.331801 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.332439 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:19.332458 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:19.332846 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:19.334721 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.334725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.334726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.334914 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.335163 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:19.335166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:19.335168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:19.335221 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:19.335569 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:19.335572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:19.335661 (+ 89us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.335749 (+ 88us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.335907 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20250905 07:54:19.336458 11177 tablet_copy_service.cc:140] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 51ad02d5fd364fd6a0e80253d6953ed0 ({username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:40002)
I20250905 07:54:19.336535 11177 tablet_copy_service.cc:161] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 51ad02d5fd364fd6a0e80253d6953ed0 at {username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:40002: session id = 51ad02d5fd364fd6a0e80253d6953ed0-00000000000000000000000000000000
I20250905 07:54:19.337322 11177 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 07:54:19.338672 11747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250905 07:54:19.341161 11747 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: tablet copy: Starting download of 0 data blocks...
I20250905 07:54:19.341369 11747 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: tablet copy: Starting download of 1 WAL segments...
I20250905 07:54:19.343197 11747 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 07:54:19.345432 11177 tablet_copy_service.cc:342] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Request end of tablet copy session 51ad02d5fd364fd6a0e80253d6953ed0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:40002
I20250905 07:54:19.345551 11177 tablet_copy_service.cc:434] P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: ending tablet copy session 51ad02d5fd364fd6a0e80253d6953ed0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 51ad02d5fd364fd6a0e80253d6953ed0
W20250905 07:54:19.350234 11747 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 07:54:19.352962 11747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:19.354727 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:19.354727 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:19.357007 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:19.357143 11747 server_base.cc:1047] running on GCE node
I20250905 07:54:19.357441 11747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:19.357784 11747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:19.359004 11747 hybrid_clock.cc:648] HybridClock initialized: now 1757058859358986 us; error 39 us; skew 500 ppm
Sep 05 07:54:19 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:19.360661 11747 init.cc:377] Logged in from keytab as oryx/127.4.49.60@KRBTEST.COM (short username oryx)
I20250905 07:54:19.361212 11747 webserver.cc:480] Webserver started at http://127.4.49.60:41297/ using document root <none> and password file <none>
I20250905 07:54:19.361402 11747 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:19.361444 11747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:19.362804 11747 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:19.363835 11948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:19.364173 11747 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:19.364254 11747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
uuid: "51ad02d5fd364fd6a0e80253d6953ed0"
format_stamp: "Formatted at 2025-09-05 07:54:19 on dist-test-slave-mnfb"
server_key: "ec334d1334f813a17f876a02d88d7764"
server_key_iv: "d4d7fe5677f16ff59847086ae984dd08"
server_key_version: "encryptionkey@0"
I20250905 07:54:19.364341 11747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:19.371995 11747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:19.374300 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.373974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:54006)
0905 07:54:19.374133 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.374198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20250905 07:54:19.374538 11155 master.cc:679] Network error: unable to get registration information for peer 51ad02d5fd364fd6a0e80253d6953ed0: Client connection negotiation failed: client connection to 127.4.49.60:45847: connect: Connection refused (error 111)
I20250905 07:54:19.375074 11747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:19.380981 11747 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.60:45847
I20250905 07:54:19.381001 12000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.60:45847 every 8 connection(s)
I20250905 07:54:19.381669 11747 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/master-2/data/info.pb
I20250905 07:54:19.383280 12001 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:54:19.384256 12001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Bootstrap starting.
I20250905 07:54:19.391309 11487 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:19.392086 12001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:54:19.392390 12001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Bootstrap complete.
I20250905 07:54:19.393098 12001 raft_consensus.cc:357] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
I20250905 07:54:19.393224 12001 raft_consensus.cc:738] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51ad02d5fd364fd6a0e80253d6953ed0, State: Initialized, Role: LEARNER
I20250905 07:54:19.393339 12001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } }
I20250905 07:54:19.393682 12004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.393788 12004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: LEARNER
I20250905 07:54:19.393838 12001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:19.396714 12016 catalog_manager.cc:1261] Loaded cluster ID: 90f4d38706574efd912be68b1e4d1b22
I20250905 07:54:19.396781 12016 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: loading cluster ID for follower catalog manager: success
I20250905 07:54:19.398694 12016 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: acquiring CA information for follower catalog manager: success
I20250905 07:54:19.400027 12016 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:54:19.445134 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.442457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:54020)
0905 07:54:19.442620 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:19.442634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:19.442723 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:19.443056 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:19.443058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:19.443071 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:19.443247 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.443252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.444177 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.444179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:19.444797 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:19.444804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.444902 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.444923 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:54:19.444958 (+ 35us) client_negotiation.cc:240] Negotiation successful
0905 07:54:19.445015 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20250905 07:54:19.445650 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:19.442570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.4.49.60:45847)
0905 07:54:19.442822 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:19.442826 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:19.442838 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:54:19.442884 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:19.442887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:19.442942 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:19.443042 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:19.443337 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.444084 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:19.444894 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:19.445238 (+ 344us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:19.445409 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:19.445462 (+ 53us) server_negotiation.cc:300] Negotiation successful
0905 07:54:19.445543 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20250905 07:54:19.615331 11625 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:19.904328 11975 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250905 07:54:19.904846 12019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:19.905006 12019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: LEARNER
I20250905 07:54:20.421720 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: NON_VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: true } } }
I20250905 07:54:20.421898 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:20.422776 12018 raft_consensus.cc:1062] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: attempting to promote NON_VOTER 51ad02d5fd364fd6a0e80253d6953ed0 to VOTER
I20250905 07:54:20.423146 12018 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } }
I20250905 07:54:20.424448 11785 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250905 07:54:20.424496 11975 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 LEARNER]: Refusing update from remote peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250905 07:54:20.424881 12020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250905 07:54:20.425103 12018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250905 07:54:20.427659 12021 raft_consensus.cc:2953] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.428280 11975 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.429334 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.429447 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:20.429440 11785 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.429618 12021 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.429703 12021 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:20.429772 12021 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.429837 12021 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:20.430814 12027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.430913 12027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:20.431041 12026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:20.431149 12026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:21.376113 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.363747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.4.49.61:33881)
0905 07:54:21.364023 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.364027 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.364134 (+ 107us) server_negotiation.cc:408] Connection header received
0905 07:54:21.364213 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.364216 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.364265 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.364343 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.365298 (+ 955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.365856 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.366839 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.366995 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.371804 (+ 4809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.371830 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.371833 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.371867 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.373880 (+ 2013us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.374596 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.374600 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.374601 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.374718 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.375057 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.375061 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.375062 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.375254 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.375388 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.375739 (+ 351us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.375887 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20250905 07:54:21.380501 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.377340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40010)
0905 07:54:21.377750 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.377764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.377872 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.378204 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.378217 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.378238 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.378457 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.378463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.379371 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.379375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.380134 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.380145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.380251 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.380272 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.380306 (+ 34us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.380369 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20250905 07:54:21.380951 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.377551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.4.49.62:43263)
0905 07:54:21.377847 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.377851 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.377867 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:21.378012 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.378016 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.378070 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.378151 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.378620 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.379233 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.380265 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.380637 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.380683 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.380750 (+ 67us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.380822 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20250905 07:54:21.385224 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.382465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51736)
0905 07:54:21.382590 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.382611 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.382705 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.382959 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.382962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.382980 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.383158 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.383163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.384278 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.384281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.384891 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.384899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.385001 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.385022 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.385056 (+ 34us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.385109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20250905 07:54:21.385749 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.382545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.4.49.61:33881)
0905 07:54:21.382661 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.382666 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.382679 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:21.382791 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.382794 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.382838 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.382909 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.383302 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.384159 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.385008 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.385485 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.385522 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.385569 (+ 47us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.385628 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250905 07:54:21.389580 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.387085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:54022)
0905 07:54:21.387218 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.387229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.387299 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.387642 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.387646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.387664 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.387834 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.387840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.388611 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.388615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.389186 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.389193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.389295 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.389318 (+ 23us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.389367 (+ 49us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.389431 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20250905 07:54:21.390002 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.387199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.4.49.60:45847)
0905 07:54:21.387444 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.387448 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.387462 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:21.387502 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.387504 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.387545 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.387618 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.387942 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.388508 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.389308 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.389770 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.389807 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.389855 (+ 48us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.389906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20250905 07:54:21.399422 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.391936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.4.49.60:45847)
0905 07:54:21.392078 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.392082 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.392097 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:21.392204 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.392206 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.392249 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.392315 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.393130 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.393624 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.394342 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.394497 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.395631 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.395653 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.395655 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.395693 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.397787 (+ 2094us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.398258 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.398262 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.398264 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.398325 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.398577 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.398580 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.398583 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.398770 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.398897 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.399120 (+ 223us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.399259 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250905 07:54:21.403726 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.400476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40022)
0905 07:54:21.400753 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.400768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.400865 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.401240 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.401243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.401264 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.401487 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.401494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.402320 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.402324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.403329 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.403342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.403472 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.403494 (+ 22us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.403531 (+ 37us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.403589 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20250905 07:54:21.404357 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.400649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.4.49.62:43263)
0905 07:54:21.400777 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.400780 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.400845 (+ 65us) server_negotiation.cc:408] Connection header received
0905 07:54:21.401058 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.401062 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.401120 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.401194 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.401667 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.402192 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.403504 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.403939 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.403989 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.404045 (+ 56us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.404145 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250905 07:54:21.410100 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.407188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.0.0.1:51750)
0905 07:54:21.407350 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.407366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.407514 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.407750 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.407753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.407774 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.407968 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.407974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.409003 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.409008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.409636 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.409646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.409786 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.409815 (+ 29us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.409868 (+ 53us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.409949 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20250905 07:54:21.410490 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.407352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.4.49.61:33881)
0905 07:54:21.407499 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.407503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.407518 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:21.407620 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.407622 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.407664 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.407736 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.408169 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.408765 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.409785 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.410232 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.410275 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.410331 (+ 56us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.410390 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20250905 07:54:21.414561 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.412020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.0.0.1:54036)
0905 07:54:21.412324 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.412338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.412412 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.412712 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.412715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.412732 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:21.412885 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.412891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.413571 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.413574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.414147 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.414155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.414266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.414290 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.414339 (+ 49us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.414403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20250905 07:54:21.414918 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.412393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.4.49.60:45847)
0905 07:54:21.412513 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.412516 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.412525 (+ 9us) server_negotiation.cc:408] Connection header received
0905 07:54:21.412555 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.412558 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.412592 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.412652 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:21.412982 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.413471 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.414262 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.414678 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.414726 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.414781 (+ 55us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.414836 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:54:21.416500 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11221
I20250905 07:54:21.422407 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:35795
--local_ip_for_outbound_sockets=127.4.49.1
--tserver_master_addrs=127.4.49.62:43263
--webserver_port=36719
--webserver_interface=127.4.49.1
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:21.543498 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:21.543776 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:21.543839 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:21.547374 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:21.547474 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:21.547567 12034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:21.552512 12034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:35795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=36719
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:21.553769 12034 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:21.554770 12034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:21.562289 12040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.562289 12042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.562289 12039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:21.562616 12034 server_base.cc:1047] running on GCE node
I20250905 07:54:21.563050 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:21.563674 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:21.564823 12034 hybrid_clock.cc:648] HybridClock initialized: now 1757058861564795 us; error 46 us; skew 500 ppm
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:21.567626 12034 init.cc:377] Logged in from keytab as oryx/127.4.49.1@KRBTEST.COM (short username oryx)
I20250905 07:54:21.568892 12034 webserver.cc:480] Webserver started at http://127.4.49.1:36719/ using document root <none> and password file <none>
I20250905 07:54:21.569479 12034 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:21.569550 12034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:21.573383 12034 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:21.575659 12049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.576990 12034 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:54:21.577142 12034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "fba28b8cfc4c466d3857db66ead34f93"
server_key_iv: "f8d5047e6ec8943f9a7c7ceb5c1f52cb"
server_key_version: "encryptionkey@0"
I20250905 07:54:21.577680 12034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:21.605594 12034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:21.608680 12034 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:21.608898 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:21.609503 12034 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:21.610476 12034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:21.610524 12034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.610565 12034 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:21.610585 12034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.619886 12034 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:35795
I20250905 07:54:21.620014 12163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:35795 every 8 connection(s)
I20250905 07:54:21.621121 12034 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:21.629281 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12034
I20250905 07:54:21.629482 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11358
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
I20250905 07:54:21.636173 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.623503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:54177 (local address 127.4.49.60:45847)
0905 07:54:21.623630 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.623634 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.624427 (+ 793us) server_negotiation.cc:408] Connection header received
0905 07:54:21.625219 (+ 792us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.625222 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.625270 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.625335 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.626939 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627632 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.628294 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628477 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.631806 (+ 3329us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.631825 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.631828 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.631857 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.633778 (+ 1921us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.634353 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.634357 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.634359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.634713 (+ 354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.635231 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.635235 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.635237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.635425 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.635508 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.635828 (+ 320us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.635964 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250905 07:54:21.636178 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.622907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:38775 (local address 127.4.49.62:43263)
0905 07:54:21.623046 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.623050 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.624041 (+ 991us) server_negotiation.cc:408] Connection header received
0905 07:54:21.625095 (+ 1054us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.625099 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.625164 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.625238 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.626891 (+ 1653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627412 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.628319 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628526 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.631417 (+ 2891us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.631447 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.631450 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.631484 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.633519 (+ 2035us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.634339 (+ 820us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.634344 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.634346 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.634410 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.634739 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.634742 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.634745 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.634972 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.635044 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.635825 (+ 781us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.635980 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:54:21.637650 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.623286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.4.49.1:54177)
0905 07:54:21.624342 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.624356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.624891 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.625434 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.625466 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.626512 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.626835 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.626843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627738 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.627742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.628181 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.628190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628411 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.629067 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.629110 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.631215 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.633916 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.633919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.633921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.634237 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.634838 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.634841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.634844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.634899 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.635576 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.635578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.635658 (+ 80us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.636387 (+ 729us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.636802 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":234,"spinlock_wait_cycles":11520,"thread_start_us":138,"threads_started":1}
I20250905 07:54:21.637656 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.623286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:38775)
0905 07:54:21.623900 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.623943 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.624881 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.625433 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.625443 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.625995 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.626638 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.626654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627537 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.627541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.628179 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.628191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628645 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.629343 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.629360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.631230 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.633698 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.633706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.633722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.634224 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.634520 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.634522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.634523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.634639 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.635119 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.635126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.635501 (+ 375us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.636369 (+ 868us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.636715 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"spinlock_wait_cycles":15232,"thread_start_us":139,"threads_started":1}
I20250905 07:54:21.637739 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.623235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:44395 (local address 127.4.49.61:33881)
0905 07:54:21.623352 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.623356 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.624471 (+ 1115us) server_negotiation.cc:408] Connection header received
0905 07:54:21.625457 (+ 986us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.625461 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.625514 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.625595 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.626796 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627552 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.628318 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628514 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.634146 (+ 5632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.634171 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.634175 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.634210 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.635954 (+ 1744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.636520 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.636524 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.636526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.636580 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.636861 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.636864 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.636865 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.637039 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.637116 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.637463 (+ 347us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.637590 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:54:21.638420 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.623286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.4.49.1:44395)
0905 07:54:21.624399 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.624410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.624891 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.625627 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.625631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.626011 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.626640 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.626655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.627658 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.627661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.628201 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.628207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.628645 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.632511 (+ 3866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.632529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.633945 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.636108 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.636111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.636114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.636407 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.636710 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.636714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.636716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.636767 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.637179 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.637182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.637260 (+ 78us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.637922 (+ 662us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.638292 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":160,"spinlock_wait_cycles":41088,"thread_start_us":119,"threads_started":1}
I20250905 07:54:21.639120 12164 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:21.639183 12166 heartbeater.cc:344] Connected to a master server at 127.4.49.61:33881
I20250905 07:54:21.639055 12165 heartbeater.cc:344] Connected to a master server at 127.4.49.60:45847
I20250905 07:54:21.639353 12166 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.639384 12165 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.639353 12164 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.640048 12164 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
I20250905 07:54:21.640048 12166 heartbeater.cc:507] Master 127.4.49.61:33881 requested a full tablet report, sending...
I20250905 07:54:21.640046 12165 heartbeater.cc:507] Master 127.4.49.60:45847 requested a full tablet report, sending...
I20250905 07:54:21.641253 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:36415
--local_ip_for_outbound_sockets=127.4.49.2
--tserver_master_addrs=127.4.49.62:43263
--webserver_port=37677
--webserver_interface=127.4.49.2
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250905 07:54:21.641433 11155 ts_manager.cc:194] Re-registered known tserver with Master: 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
I20250905 07:54:21.642031 11155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:38775
I20250905 07:54:21.642128 11775 ts_manager.cc:194] Registered new tserver with Master: 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
I20250905 07:54:21.642916 11965 ts_manager.cc:194] Registered new tserver with Master: 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
W20250905 07:54:21.752236 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:21.752485 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:21.752557 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:21.756044 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:21.756170 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:21.756273 12171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:21.760926 12171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:36415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=37677
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:21.762068 12171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:21.762919 12171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:21.769708 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.769708 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.769707 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:21.769948 12171 server_base.cc:1047] running on GCE node
I20250905 07:54:21.770328 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:21.770946 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:21.772128 12171 hybrid_clock.cc:648] HybridClock initialized: now 1757058861772109 us; error 33 us; skew 500 ppm
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:21.774979 12171 init.cc:377] Logged in from keytab as oryx/127.4.49.2@KRBTEST.COM (short username oryx)
I20250905 07:54:21.776178 12171 webserver.cc:480] Webserver started at http://127.4.49.2:37677/ using document root <none> and password file <none>
I20250905 07:54:21.776795 12171 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:21.776861 12171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:21.780776 12171 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20250905 07:54:21.782939 12186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.784137 12171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:21.784251 12171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "477f5d65c7ca4465ac2287c067ffbfdc"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "4b1ed45adb8826299a5e6ab6be4a88c8"
server_key_iv: "d5cc17621eb90173401d36a83f546c8b"
server_key_version: "encryptionkey@0"
I20250905 07:54:21.784695 12171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:21.797859 12171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:21.800944 12171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:21.801120 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:21.801671 12171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:21.802609 12171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:21.802659 12171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.802701 12171 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:21.802724 12171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.811681 12171 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:36415
I20250905 07:54:21.811712 12299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:36415 every 8 connection(s)
I20250905 07:54:21.812825 12171 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:21.818315 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12171
I20250905 07:54:21.818499 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11494
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:21.827411 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:37791
--local_ip_for_outbound_sockets=127.4.49.3
--tserver_master_addrs=127.4.49.62:43263
--webserver_port=39135
--webserver_interface=127.4.49.3
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847 with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250905 07:54:21.827742 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.815167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:59733 (local address 127.4.49.61:33881)
0905 07:54:21.815294 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.815298 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.816277 (+ 979us) server_negotiation.cc:408] Connection header received
0905 07:54:21.817008 (+ 731us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.817012 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.817072 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.817143 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.819364 (+ 2221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.819945 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.820678 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.820855 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.823590 (+ 2735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.823611 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.823614 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.823642 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.825442 (+ 1800us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.825998 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.826002 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.826004 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.826058 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.826378 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.826381 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.826382 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.826588 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.826685 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.827444 (+ 759us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.827580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250905 07:54:21.827978 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.815121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:43517 (local address 127.4.49.62:43263)
0905 07:54:21.815418 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.815423 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.815921 (+ 498us) server_negotiation.cc:408] Connection header received
0905 07:54:21.817034 (+ 1113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.817038 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.817097 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.817146 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.820973 (+ 3827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.821756 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.822686 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.822892 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.824805 (+ 1913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.824826 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.824829 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.824859 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.826365 (+ 1506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.826785 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.826789 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.826791 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.826843 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.827099 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.827102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.827104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.827286 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.827357 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.827691 (+ 334us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.827820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20250905 07:54:21.829659 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.814911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.4.49.2:59733)
0905 07:54:21.816193 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.816204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.816801 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.817320 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.817328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.817815 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.819178 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.819198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.820128 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.820131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.820558 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.820565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.820802 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.821445 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.821480 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.823423 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.825577 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.825583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.825599 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.825898 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.826188 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.826191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.826193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.826292 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.826756 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.826765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.827130 (+ 365us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.828442 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.828719 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":385,"spinlock_wait_cycles":12160,"thread_start_us":100,"threads_started":1}
I20250905 07:54:21.830925 12302 heartbeater.cc:344] Connected to a master server at 127.4.49.61:33881
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
I20250905 07:54:21.831204 12302 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.831796 12302 heartbeater.cc:507] Master 127.4.49.61:33881 requested a full tablet report, sending...
I20250905 07:54:21.832444 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.814911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:43517)
0905 07:54:21.815761 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.815806 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.816795 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.817321 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.817330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.817830 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.820805 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.820818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.821911 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.821914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.822519 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.822529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.822678 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.823181 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.823197 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.824667 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.826485 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.826488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.826489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.826680 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.826942 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.826946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.826949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.827004 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.827407 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.827410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.827522 (+ 112us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.828218 (+ 696us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.832270 (+ 4052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"spinlock_wait_cycles":19456,"thread_start_us":115,"threads_started":1}
I20250905 07:54:21.833088 11775 ts_manager.cc:194] Registered new tserver with Master: 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
I20250905 07:54:21.833323 12300 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:21.833441 12300 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.833689 12300 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
I20250905 07:54:21.834406 11155 ts_manager.cc:194] Re-registered known tserver with Master: 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
I20250905 07:54:21.835198 11155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:43517
I20250905 07:54:21.837050 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.815261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:51173 (local address 127.4.49.60:45847)
0905 07:54:21.815526 (+ 265us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:21.815531 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:21.815967 (+ 436us) server_negotiation.cc:408] Connection header received
0905 07:54:21.817035 (+ 1068us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:21.817038 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:21.817095 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:21.817146 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:21.827186 (+ 10040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.827984 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.829150 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.829342 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.833322 (+ 3980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:21.833347 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:21.833350 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:21.833388 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:21.835390 (+ 2002us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.835985 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.835989 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.835991 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.836044 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:21.836303 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:21.836307 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:21.836309 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:21.836471 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:21.836556 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:21.836802 (+ 246us) server_negotiation.cc:300] Negotiation successful
0905 07:54:21.836914 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20250905 07:54:21.837070 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:21.814911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.4.49.2:51173)
0905 07:54:21.815897 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:21.815908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:21.816798 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:21.817321 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:21.817330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:21.818251 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:21.827040 (+ 8789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.827049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.828423 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:21.828426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:21.829011 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:21.829020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:21.829470 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:21.830051 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:21.830067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:21.833153 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:21.835681 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.835685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.835686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.835867 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.836164 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:21.836167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:21.836168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:21.836216 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:21.836557 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:21.836560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:21.836639 (+ 79us) client_negotiation.cc:769] Sending connection context
0905 07:54:21.836724 (+ 85us) client_negotiation.cc:240] Negotiation successful
0905 07:54:21.836875 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":47,"spinlock_wait_cycles":17664,"thread_start_us":125,"threads_started":1}
I20250905 07:54:21.838053 12301 heartbeater.cc:344] Connected to a master server at 127.4.49.60:45847
I20250905 07:54:21.838143 12301 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:21.838344 12301 heartbeater.cc:507] Master 127.4.49.60:45847 requested a full tablet report, sending...
I20250905 07:54:21.839053 11965 ts_manager.cc:194] Registered new tserver with Master: 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
W20250905 07:54:21.944248 12307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:21.944537 12307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:21.944623 12307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:21.948386 12307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:21.948468 12307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:21.948555 12307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:21.953094 12307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:37791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=39135
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:43263,127.4.49.61:33881,127.4.49.60:45847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:21.954305 12307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:21.955227 12307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:21.962352 12315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.962404 12312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:21.962352 12313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:21.962638 12307 server_base.cc:1047] running on GCE node
I20250905 07:54:21.963122 12307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:21.963733 12307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:21.964884 12307 hybrid_clock.cc:648] HybridClock initialized: now 1757058861964854 us; error 41 us; skew 500 ppm
Sep 05 07:54:21 dist-test-slave-mnfb krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:21.967960 12307 init.cc:377] Logged in from keytab as oryx/127.4.49.3@KRBTEST.COM (short username oryx)
I20250905 07:54:21.969190 12307 webserver.cc:480] Webserver started at http://127.4.49.3:39135/ using document root <none> and password file <none>
I20250905 07:54:21.969808 12307 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:21.969883 12307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:21.974009 12307 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:21.976395 12322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.977622 12307 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:54:21.977752 12307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "d83ef27ae9e547a1afe043b7a0b956ea"
format_stamp: "Formatted at 2025-09-05 07:54:18 on dist-test-slave-mnfb"
server_key: "f08ea9c07e66d1a4565b388dc2e68b90"
server_key_iv: "f042d30e7653366dbbcd745802fda199"
server_key_version: "encryptionkey@0"
I20250905 07:54:21.978192 12307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:21.994551 12307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:21.997479 12307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:21.997684 12307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:21.998299 12307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:21.999217 12307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:21.999264 12307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:21.999305 12307 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:21.999325 12307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:22.008572 12307 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:37791
I20250905 07:54:22.008602 12438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:37791 every 8 connection(s)
I20250905 07:54:22.009773 12307 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:22.015918 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12307
Sep 05 07:54:22 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
Sep 05 07:54:22 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.61@KRBTEST.COM
Sep 05 07:54:22 dist-test-slave-mnfb krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.60@KRBTEST.COM
I20250905 07:54:22.027113 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.011593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:36329 (local address 127.4.49.61:33881)
0905 07:54:22.011753 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.011758 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.013221 (+ 1463us) server_negotiation.cc:408] Connection header received
0905 07:54:22.014299 (+ 1078us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.014304 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.014361 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.014464 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.016534 (+ 2070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.017239 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.018186 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.018374 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.023004 (+ 4630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.023023 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.023026 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.023054 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.025021 (+ 1967us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025418 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.025421 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.025423 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.025476 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.026122 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.026126 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.026128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.026291 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.026373 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.026817 (+ 444us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.026940 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250905 07:54:22.027765 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.011882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:37743 (local address 127.4.49.62:43263)
0905 07:54:22.012032 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.012036 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.012714 (+ 678us) server_negotiation.cc:408] Connection header received
0905 07:54:22.014949 (+ 2235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.014953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.015011 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.015085 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.016536 (+ 1451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.017312 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.018216 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.018421 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.021858 (+ 3437us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.021880 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.021883 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.021911 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.024196 (+ 2285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025215 (+ 1019us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.025219 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.025221 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.025276 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025584 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.025587 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.025589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.025791 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.025878 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.027446 (+ 1568us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.027594 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250905 07:54:22.028003 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.012113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.60:45847 (local address 127.4.49.3:41881)
0905 07:54:22.014179 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.014192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.014348 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.014738 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.014748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.017829 (+ 3081us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:22.018112 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.018119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.019057 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.019060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.019528 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.019537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.020001 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.020740 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:22.020752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:22.022533 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:22.025424 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.025429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.025601 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.026282 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.026286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.026287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.026335 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.026742 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:22.026744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:22.026814 (+ 70us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.027134 (+ 320us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.027360 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1994,"mutex_wait_us":136,"thread_start_us":105,"threads_started":1}
I20250905 07:54:22.028115 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.012009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:37743)
0905 07:54:22.012543 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.012591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.014460 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.015200 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.015203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.015478 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:22.016346 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.016361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.017495 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.017498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.018086 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.018094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.018648 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.019622 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:22.019666 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:22.021705 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:22.024299 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.024306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.024324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.024692 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.025347 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.025352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.025501 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.025873 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:22.025880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:22.026204 (+ 324us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.026501 (+ 297us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.026830 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"spinlock_wait_cycles":22528,"thread_start_us":137,"threads_started":1}
I20250905 07:54:22.028368 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.012168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:41881 (local address 127.4.49.60:45847)
0905 07:54:22.012295 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.012300 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.014273 (+ 1973us) server_negotiation.cc:408] Connection header received
0905 07:54:22.014464 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.014468 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.014519 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.014570 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.018259 (+ 3689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.018921 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.019682 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.019875 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.022670 (+ 2795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.022694 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.022697 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.022733 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.025089 (+ 2356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025947 (+ 858us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.025950 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.025952 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.026011 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.026427 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.026430 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.026432 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.026611 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.026713 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.028083 (+ 1370us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.028223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250905 07:54:22.028008 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.012009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.61:33881 (local address 127.4.49.3:36329)
0905 07:54:22.013094 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.013108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.014034 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.014737 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.014749 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.015470 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:22.016341 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.016361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.017447 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.017451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.018043 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.018052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.018648 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.020287 (+ 1639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:22.020303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:22.022866 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:22.025134 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.025140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.025329 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.025689 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:22.025692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:22.025694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:22.025738 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:22.026479 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:22.026482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:22.026565 (+ 83us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.026657 (+ 92us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.027309 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":562,"spinlock_wait_cycles":26880,"thread_start_us":135,"threads_started":1}
I20250905 07:54:22.029238 12439 heartbeater.cc:344] Connected to a master server at 127.4.49.62:43263
I20250905 07:54:22.029271 12441 heartbeater.cc:344] Connected to a master server at 127.4.49.61:33881
I20250905 07:54:22.029513 12439 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:22.029513 12441 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:22.030119 12439 heartbeater.cc:507] Master 127.4.49.62:43263 requested a full tablet report, sending...
I20250905 07:54:22.030117 12441 heartbeater.cc:507] Master 127.4.49.61:33881 requested a full tablet report, sending...
I20250905 07:54:22.030699 12440 heartbeater.cc:344] Connected to a master server at 127.4.49.60:45847
I20250905 07:54:22.030791 12440 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:22.030998 12440 heartbeater.cc:507] Master 127.4.49.60:45847 requested a full tablet report, sending...
I20250905 07:54:22.031339 11775 ts_manager.cc:194] Registered new tserver with Master: d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791)
I20250905 07:54:22.031742 11965 ts_manager.cc:194] Registered new tserver with Master: d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791)
I20250905 07:54:22.031891 11155 ts_manager.cc:194] Re-registered known tserver with Master: d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791)
I20250905 07:54:22.032810 11155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:37743
I20250905 07:54:22.035334 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.019365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.4.49.62:43263)
0905 07:54:22.021356 (+ 1991us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.021360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.021375 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:22.021420 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.021424 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.021492 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.021576 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.024156 (+ 2580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.024980 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.028070 (+ 3090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.029149 (+ 1079us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.030859 (+ 1710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.030879 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.030883 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.030912 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.033805 (+ 2893us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.034183 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.034186 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.034188 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.034241 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.034472 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.034476 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.034478 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.034658 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.034783 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.035020 (+ 237us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.035150 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895,"thread_start_us":97,"threads_started":1}
I20250905 07:54:22.035730 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.019569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.4.49.61:33881)
0905 07:54:22.020482 (+ 913us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.020487 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.021033 (+ 546us) server_negotiation.cc:408] Connection header received
0905 07:54:22.021124 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.021127 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.021865 (+ 738us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.021942 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.024120 (+ 2178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.025717 (+ 1597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.028740 (+ 3023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.029387 (+ 647us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.031005 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.031020 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.031023 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.031050 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.033067 (+ 2017us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.033717 (+ 650us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.033721 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.033723 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.033776 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.034036 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.034039 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.034041 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.034218 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.034340 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.034877 (+ 537us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.035572 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":57,"threads_started":1}
I20250905 07:54:22.038367 11155 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:22.041494 11155 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:22.042246 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.023364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.4.49.60:45847)
0905 07:54:22.028541 (+ 5177us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.028545 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.034760 (+ 6215us) server_negotiation.cc:408] Connection header received
0905 07:54:22.034810 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.034813 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.034871 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.034935 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.036086 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.036757 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.037614 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.037818 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.038931 (+ 1113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.038954 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.038957 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.038990 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.040640 (+ 1650us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.041088 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.041092 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.041094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.041148 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.041339 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.041341 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.041343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.041537 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.041674 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.041933 (+ 259us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.042072 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5106,"thread_start_us":90,"threads_started":1}
I20250905 07:54:22.058584 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.053837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:35795 (local address 127.0.0.1:39842)
0905 07:54:22.054318 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.054330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.054405 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.056159 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.056163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.056212 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:22.056484 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.056493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.057540 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.057544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.058150 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.058157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.058258 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.058285 (+ 27us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.058333 (+ 48us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.058403 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"spinlock_wait_cycles":1408,"thread_start_us":81,"threads_started":1}
I20250905 07:54:22.059163 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.053893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:36415 (local address 127.0.0.1:37834)
0905 07:54:22.055260 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.055272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.055399 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.056028 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.056032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.056116 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:22.056487 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.056496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.057956 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.057959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.058855 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.058865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.058947 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.058968 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.059004 (+ 36us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.059056 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"spinlock_wait_cycles":2304,"thread_start_us":85,"threads_started":1}
I20250905 07:54:22.059274 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.054469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.4.49.1:35795)
0905 07:54:22.054860 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.054868 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.054893 (+ 25us) server_negotiation.cc:408] Connection header received
0905 07:54:22.054986 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.054993 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.055165 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.055315 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:22.056639 (+ 1324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.057428 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.058288 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.058823 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.058976 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.059045 (+ 69us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.059132 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20250905 07:54:22.060106 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.053665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37791 (local address 127.0.0.1:47636)
0905 07:54:22.053985 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.054001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.054132 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.056484 (+ 2352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.056487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.056507 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:22.056705 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.056713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.058895 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.058899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.059710 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.059719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.059821 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.059839 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.059881 (+ 42us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.059937 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
I20250905 07:54:22.061015 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.054617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.4.49.2:36415)
0905 07:54:22.054948 (+ 331us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.054953 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.055374 (+ 421us) server_negotiation.cc:408] Connection header received
0905 07:54:22.055569 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.055574 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.055864 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.056649 (+ 785us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:22.056802 (+ 153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.057823 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.059499 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.060576 (+ 1077us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.060711 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.060781 (+ 70us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.060869 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":91,"threads_started":1}
I20250905 07:54:22.061015 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.053840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.4.49.3:37791)
0905 07:54:22.055710 (+ 1870us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.055717 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.055739 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:54:22.055820 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.055825 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.055998 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.056145 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:22.057334 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.058466 (+ 1132us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.059863 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.060603 (+ 740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.060710 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.060781 (+ 71us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.060873 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":130,"threads_started":1}
I20250905 07:54:22.062492 12097 tablet_service.cc:1468] Processing CreateTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 (DEFAULT_TABLE table=test-table [id=0773a0323b3a4fa09a200647c20bbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:22.063470 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4020dbb2374473bb9e28d7d4609b3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:22.063933 12372 tablet_service.cc:1468] Processing CreateTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 (DEFAULT_TABLE table=test-table [id=0773a0323b3a4fa09a200647c20bbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:22.063933 12234 tablet_service.cc:1468] Processing CreateTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 (DEFAULT_TABLE table=test-table [id=0773a0323b3a4fa09a200647c20bbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:22.064987 12372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4020dbb2374473bb9e28d7d4609b3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:22.064986 12234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4020dbb2374473bb9e28d7d4609b3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:22.070432 12473 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Bootstrap starting.
I20250905 07:54:22.073222 12475 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Bootstrap starting.
I20250905 07:54:22.073338 12474 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Bootstrap starting.
I20250905 07:54:22.074292 12473 tablet_bootstrap.cc:654] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:22.075270 12473 log.cc:826] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:22.075649 12474 tablet_bootstrap.cc:654] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:22.075753 12475 tablet_bootstrap.cc:654] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:22.076449 12474 log.cc:826] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:22.076520 12475 log.cc:826] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:22.077471 12473 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: No bootstrap required, opened a new log
I20250905 07:54:22.077692 12473 ts_tablet_manager.cc:1397] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250905 07:54:22.078312 12475 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: No bootstrap required, opened a new log
I20250905 07:54:22.078341 12474 tablet_bootstrap.cc:492] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: No bootstrap required, opened a new log
I20250905 07:54:22.078500 12475 ts_tablet_manager.cc:1397] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:54:22.078512 12474 ts_tablet_manager.cc:1397] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:54:22.080982 12475 raft_consensus.cc:357] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.080988 12473 raft_consensus.cc:357] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.081193 12475 raft_consensus.cc:383] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:22.081202 12473 raft_consensus.cc:383] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:22.081250 12475 raft_consensus.cc:738] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83ef27ae9e547a1afe043b7a0b956ea, State: Initialized, Role: FOLLOWER
I20250905 07:54:22.081254 12473 raft_consensus.cc:738] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec78dd4dd4538b0d5fb795790f03f, State: Initialized, Role: FOLLOWER
I20250905 07:54:22.081424 12474 raft_consensus.cc:357] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.081691 12474 raft_consensus.cc:383] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:22.081736 12474 raft_consensus.cc:738] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477f5d65c7ca4465ac2287c067ffbfdc, State: Initialized, Role: FOLLOWER
I20250905 07:54:22.081734 12473 consensus_queue.cc:260] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.081736 12475 consensus_queue.cc:260] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.082127 12474 consensus_queue.cc:260] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.082741 12475 ts_tablet_manager.cc:1428] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250905 07:54:22.082849 12473 ts_tablet_manager.cc:1428] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:54:22.082870 12300 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:22.083092 12439 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:22.083173 12164 heartbeater.cc:499] Master 127.4.49.62:43263 was elected leader, sending a full tablet report...
I20250905 07:54:22.085937 12474 ts_tablet_manager.cc:1428] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20250905 07:54:22.122563 12167 tablet.cc:2378] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:54:22.125877 12481 raft_consensus.cc:491] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:22.126084 12481 raft_consensus.cc:513] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.127357 12481 leader_election.cc:290] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791), 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
I20250905 07:54:22.130276 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.127576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37791 (local address 127.4.49.2:33731)
0905 07:54:22.127742 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.127759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.127853 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.128204 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.128206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.128223 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:22.128448 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.128458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.129184 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.129188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.129869 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.129879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.129987 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.130011 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.130061 (+ 50us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.130130 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20250905 07:54:22.130687 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.127576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:35795 (local address 127.4.49.2:41593)
0905 07:54:22.127965 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:22.127979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:22.128097 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:22.128410 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:22.128413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:22.128433 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:22.128631 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.128638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.129355 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.129358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:22.130032 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:22.130041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.130160 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.130180 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:54:22.130227 (+ 47us) client_negotiation.cc:240] Negotiation successful
0905 07:54:22.130286 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":212}
I20250905 07:54:22.130707 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.127809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:33731 (local address 127.4.49.3:37791)
0905 07:54:22.127945 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.127950 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.127965 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:22.128019 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.128022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.128127 (+ 105us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.128195 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:22.128573 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.129067 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.129985 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.130455 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.130500 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.130551 (+ 51us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.130615 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:22.130900 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.128038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:41593 (local address 127.4.49.1:35795)
0905 07:54:22.128201 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.128205 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.128217 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:54:22.128261 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.128263 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.128304 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.128372 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:22.128759 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.129243 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.130158 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.130627 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.130667 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.130729 (+ 62us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.130794 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20250905 07:54:22.131350 12392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4020dbb2374473bb9e28d7d4609b3f1" candidate_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" is_pre_election: true
I20250905 07:54:22.131431 12117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4020dbb2374473bb9e28d7d4609b3f1" candidate_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" is_pre_election: true
I20250905 07:54:22.131665 12392 raft_consensus.cc:2466] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477f5d65c7ca4465ac2287c067ffbfdc in term 0.
I20250905 07:54:22.131690 12117 raft_consensus.cc:2466] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477f5d65c7ca4465ac2287c067ffbfdc in term 0.
I20250905 07:54:22.132192 12188 leader_election.cc:304] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec78dd4dd4538b0d5fb795790f03f, 477f5d65c7ca4465ac2287c067ffbfdc; no voters:
I20250905 07:54:22.132483 12481 raft_consensus.cc:2802] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:22.132558 12481 raft_consensus.cc:491] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:22.132589 12481 raft_consensus.cc:3058] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:22.133642 12481 raft_consensus.cc:513] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.134079 12481 leader_election.cc:290] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [CANDIDATE]: Term 1 election: Requested vote from peers d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791), 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795)
I20250905 07:54:22.134544 12392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4020dbb2374473bb9e28d7d4609b3f1" candidate_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83ef27ae9e547a1afe043b7a0b956ea"
I20250905 07:54:22.134562 12117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4020dbb2374473bb9e28d7d4609b3f1" candidate_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
I20250905 07:54:22.134701 12117 raft_consensus.cc:3058] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:22.134714 12392 raft_consensus.cc:3058] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:22.136005 12392 raft_consensus.cc:2466] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477f5d65c7ca4465ac2287c067ffbfdc in term 1.
I20250905 07:54:22.136005 12117 raft_consensus.cc:2466] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477f5d65c7ca4465ac2287c067ffbfdc in term 1.
I20250905 07:54:22.136487 12190 leader_election.cc:304] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477f5d65c7ca4465ac2287c067ffbfdc, d83ef27ae9e547a1afe043b7a0b956ea; no voters:
I20250905 07:54:22.136776 12481 raft_consensus.cc:2802] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:22.136963 12481 raft_consensus.cc:695] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 LEADER]: Becoming Leader. State: Replica: 477f5d65c7ca4465ac2287c067ffbfdc, State: Running, Role: LEADER
I20250905 07:54:22.137324 12481 consensus_queue.cc:237] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } }
I20250905 07:54:22.141396 11156 catalog_manager.cc:5582] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2). New cstate: current_term: 1 leader_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } health_report { overall_health: UNKNOWN } } }
I20250905 07:54:22.170699 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.167738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.4.49.2:36415)
0905 07:54:22.167862 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.167865 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.167877 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:54:22.167958 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.167961 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.168018 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.168126 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:22.168521 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.169015 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.169686 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.169828 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.169923 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:22.170372 (+ 449us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:22.170472 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.170523 (+ 51us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.170575 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:54:22.177686 12392 raft_consensus.cc:1273] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Refusing update from remote peer 477f5d65c7ca4465ac2287c067ffbfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:22.177685 12117 raft_consensus.cc:1273] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Refusing update from remote peer 477f5d65c7ca4465ac2287c067ffbfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:22.178529 12481 consensus_queue.cc:1035] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:22.183943 12481 consensus_queue.cc:1035] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:22.194358 12486 mvcc.cc:204] Tried to move back new op lower bound from 7196913099470684160 to 7196913099317551104. Current Snapshot: MvccSnapshot[applied={T|T < 7196913099470684160 or (T in {7196913099470684160})}]
I20250905 07:54:22.196005 11156 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40030:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:22.196206 11156 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40030:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:22.203711 11156 catalog_manager.cc:5869] T 00000000000000000000000000000000 P aa5f3ac0b28c4ff9ad224ec8ae03a2cd: Sending DeleteTablet for 3 replicas of tablet e4020dbb2374473bb9e28d7d4609b3f1
I20250905 07:54:22.204471 12372 tablet_service.cc:1515] Processing DeleteTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:22 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:47636
I20250905 07:54:22.204571 12097 tablet_service.cc:1515] Processing DeleteTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:22 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:39842
I20250905 07:54:22.205672 12497 tablet_replica.cc:331] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: stopping tablet replica
I20250905 07:54:22.205729 12234 tablet_service.cc:1515] Processing DeleteTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:22 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:37834
I20250905 07:54:22.205931 12497 raft_consensus.cc:2241] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:22.206207 12233 tablet_service.cc:1515] Processing DeleteTablet for tablet e4020dbb2374473bb9e28d7d4609b3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:22 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:37834
I20250905 07:54:22.206243 12497 raft_consensus.cc:2270] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:22.206264 12498 tablet_replica.cc:331] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: stopping tablet replica
I20250905 07:54:22.206487 12498 raft_consensus.cc:2241] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:22.206703 12495 tablet_replica.cc:331] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: stopping tablet replica
I20250905 07:54:22.206866 12498 raft_consensus.cc:2270] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:22.208088 12495 raft_consensus.cc:2241] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:22.208416 12495 raft_consensus.cc:2270] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:22.209585 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11124
I20250905 07:54:22.210824 12495 ts_tablet_manager.cc:1905] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:22.213061 12495 ts_tablet_manager.cc:1918] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:22.213140 12495 log.cc:1199] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/e4020dbb2374473bb9e28d7d4609b3f1
I20250905 07:54:22.213369 12495 ts_tablet_manager.cc:1939] T e4020dbb2374473bb9e28d7d4609b3f1 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting consensus metadata
I20250905 07:54:22.215812 12497 ts_tablet_manager.cc:1905] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 07:54:22.217252 12326 connection.cc:537] server connection from 127.0.0.1:47636 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 07:54:22.218545 12497 ts_tablet_manager.cc:1918] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:22.218657 12497 log.cc:1199] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/e4020dbb2374473bb9e28d7d4609b3f1
I20250905 07:54:22.218973 12497 ts_tablet_manager.cc:1939] T e4020dbb2374473bb9e28d7d4609b3f1 P 1cfec78dd4dd4538b0d5fb795790f03f: Deleting consensus metadata
W20250905 07:54:22.220032 12052 connection.cc:441] server connection from 127.0.0.1:39842 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39842 (request call id 1) could send its response
I20250905 07:54:22.221419 12498 ts_tablet_manager.cc:1905] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:22.223877 12498 ts_tablet_manager.cc:1918] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:22.223991 12498 log.cc:1199] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/e4020dbb2374473bb9e28d7d4609b3f1
I20250905 07:54:22.224323 12498 ts_tablet_manager.cc:1939] T e4020dbb2374473bb9e28d7d4609b3f1 P 477f5d65c7ca4465ac2287c067ffbfdc: Deleting consensus metadata
W20250905 07:54:22.225517 12187 connection.cc:441] server connection from 127.0.0.1:37834 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37834 (request call id 1) could send its response
W20250905 07:54:22.225828 12187 connection.cc:441] server connection from 127.0.0.1:37834 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37834 (request call id 2) could send its response
I20250905 07:54:22.230124 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.221048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.4.49.60:45847)
0905 07:54:22.221167 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.221172 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.221421 (+ 249us) server_negotiation.cc:408] Connection header received
0905 07:54:22.221655 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.221659 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.221716 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.221799 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.222807 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.223591 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.224713 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.224912 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.226921 (+ 2009us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.226944 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.226947 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.226982 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.228609 (+ 1627us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.228958 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.228962 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.228964 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.229014 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.229242 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.229246 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.229249 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.229446 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.229581 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.229832 (+ 251us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.229953 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250905 07:54:22.231449 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:22.221509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.4.49.61:33881)
0905 07:54:22.221623 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:22.221627 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:22.222203 (+ 576us) server_negotiation.cc:408] Connection header received
0905 07:54:22.222316 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:22.222319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:22.222371 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:22.222446 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:22.224521 (+ 2075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.225459 (+ 938us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:22.226573 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:22.226791 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:22.228152 (+ 1361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:22.228174 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:22.228178 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:22.228210 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:22.229970 (+ 1760us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.230441 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.230444 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.230446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.230490 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:22.230709 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:22.230711 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:22.230713 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:22.230858 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:22.230973 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:22.231173 (+ 200us) server_negotiation.cc:300] Negotiation successful
0905 07:54:22.231300 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250905 07:54:23.194928 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.194405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:55387)
0905 07:54:23.194717 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.194806 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
W20250905 07:54:23.195292 12164 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:43263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.206355 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.205921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:43193)
0905 07:54:23.206168 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.206254 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
W20250905 07:54:23.206629 12439 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:43263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.210278 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.209842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:58557)
0905 07:54:23.210107 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.210184 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
W20250905 07:54:23.210541 12300 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:43263 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.704098 12508 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd)
I20250905 07:54:23.704289 12508 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } }
I20250905 07:54:23.705281 12508 leader_election.cc:290] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263), 51ad02d5fd364fd6a0e80253d6953ed0 (127.4.49.60:45847)
I20250905 07:54:23.705793 11975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ba70538fb1d4086953b29acbf711580" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" is_pre_election: true
I20250905 07:54:23.705875 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.705477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40472)
0905 07:54:23.705716 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.705783 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
W20250905 07:54:23.706130 11761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.706993 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.706774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40480)
0905 07:54:23.706885 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.706932 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20250905 07:54:23.707106 11761 leader_election.cc:336] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263): Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.707180 11761 leader_election.cc:304] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ba70538fb1d4086953b29acbf711580; no voters: 51ad02d5fd364fd6a0e80253d6953ed0, aa5f3ac0b28c4ff9ad224ec8ae03a2cd
I20250905 07:54:23.707450 12508 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 07:54:23.708549 12511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd)
I20250905 07:54:23.708706 12511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } }
I20250905 07:54:23.709578 12511 leader_election.cc:290] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263), 0ba70538fb1d4086953b29acbf711580 (127.4.49.61:33881)
I20250905 07:54:23.710040 11785 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ba70538fb1d4086953b29acbf711580" is_pre_election: true
I20250905 07:54:23.710166 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.709776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40486)
0905 07:54:23.709994 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.710070 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20250905 07:54:23.710235 11785 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ad02d5fd364fd6a0e80253d6953ed0 in term 1.
W20250905 07:54:23.710353 11951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.710650 11952 leader_election.cc:304] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba70538fb1d4086953b29acbf711580, 51ad02d5fd364fd6a0e80253d6953ed0; no voters:
I20250905 07:54:23.710879 12511 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 07:54:23.710945 12511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa5f3ac0b28c4ff9ad224ec8ae03a2cd)
I20250905 07:54:23.710994 12511 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:54:23.711187 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.710958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40494)
0905 07:54:23.711080 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.711126 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20250905 07:54:23.711292 11951 leader_election.cc:336] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263): Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.712141 12511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } }
I20250905 07:54:23.712514 12511 leader_election.cc:290] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 election: Requested vote from peers aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263), 0ba70538fb1d4086953b29acbf711580 (127.4.49.61:33881)
I20250905 07:54:23.712895 11785 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ba70538fb1d4086953b29acbf711580"
I20250905 07:54:23.712949 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.712631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40510)
0905 07:54:23.712838 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.712888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20250905 07:54:23.713011 11785 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:54:23.713706 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.713402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40520)
0905 07:54:23.713592 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.713649 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20250905 07:54:23.713822 11951 leader_election.cc:336] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263): Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
I20250905 07:54:23.714015 11785 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ad02d5fd364fd6a0e80253d6953ed0 in term 2.
I20250905 07:54:23.714283 11952 leader_election.cc:304] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ba70538fb1d4086953b29acbf711580, 51ad02d5fd364fd6a0e80253d6953ed0; no voters: aa5f3ac0b28c4ff9ad224ec8ae03a2cd
I20250905 07:54:23.714557 12511 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 07:54:23.714742 12511 raft_consensus.cc:695] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [term 2 LEADER]: Becoming Leader. State: Replica: 51ad02d5fd364fd6a0e80253d6953ed0, State: Running, Role: LEADER
I20250905 07:54:23.714979 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } }
I20250905 07:54:23.716027 12514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ad02d5fd364fd6a0e80253d6953ed0. Latest consensus state: current_term: 2 leader_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:23.716169 12514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:23.716460 12516 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:23.718572 12516 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:23.718968 12516 catalog_manager.cc:1261] Loaded cluster ID: 90f4d38706574efd912be68b1e4d1b22
I20250905 07:54:23.719013 12516 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:23.719615 12516 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:23.720008 12516 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Loaded TSK: 0
I20250905 07:54:23.720736 12516 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:54:23.739916 11965 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:23.742334 11965 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:23.745497 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.745213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40526)
0905 07:54:23.745334 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.745400 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20250905 07:54:23.745545 11785 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [term 2 FOLLOWER]: Refusing update from remote peer 51ad02d5fd364fd6a0e80253d6953ed0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250905 07:54:23.746075 12514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250905 07:54:23.746197 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.745982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.0.0.1:40538)
0905 07:54:23.746087 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.746140 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250905 07:54:23.746302 11951 consensus_peers.cc:489] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 -> Peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd (127.4.49.62:43263): Couldn't send request to peer aa5f3ac0b28c4ff9ad224ec8ae03a2cd. Status: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:54:23.747530 12508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ad02d5fd364fd6a0e80253d6953ed0. Latest consensus state: current_term: 2 leader_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:23.747639 12508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:23.748698 12511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:23.748817 12511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:23.749356 12508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aa5f3ac0b28c4ff9ad224ec8ae03a2cd" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 43263 } } peers { permanent_uuid: "0ba70538fb1d4086953b29acbf711580" member_type: VOTER last_known_addr { host: "127.4.49.61" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "51ad02d5fd364fd6a0e80253d6953ed0" member_type: VOTER last_known_addr { host: "127.4.49.60" port: 45847 } attrs { promote: false } } }
I20250905 07:54:23.749508 12508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba70538fb1d4086953b29acbf711580 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:54:23.758749 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.755118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:37791 (local address 127.0.0.1:47650)
0905 07:54:23.755287 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.755303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:23.755421 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:23.756340 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:23.756343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:23.756364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:23.756618 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.756625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.757497 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.757500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:23.758333 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.758341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.758455 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.758479 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:54:23.758527 (+ 48us) client_negotiation.cc:240] Negotiation successful
0905 07:54:23.758594 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20250905 07:54:23.759248 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.755232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.4.49.3:37791)
0905 07:54:23.756117 (+ 885us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.756121 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.756137 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:23.756190 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.756194 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.756249 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.756330 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:23.756714 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.757362 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.758464 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.758958 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.758997 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.759050 (+ 53us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.759115 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":97,"threads_started":1}
I20250905 07:54:23.759572 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.755624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:35795 (local address 127.0.0.1:39852)
0905 07:54:23.755899 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.755913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:23.755997 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:23.756899 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:23.756902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:23.756920 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:23.757138 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.757146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.758007 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.758010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:23.759051 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.759061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.759152 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.759169 (+ 17us) client_negotiation.cc:769] Sending connection context
0905 07:54:23.759212 (+ 43us) client_negotiation.cc:240] Negotiation successful
0905 07:54:23.759269 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
I20250905 07:54:23.759794 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.755642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:36415 (local address 127.0.0.1:37850)
0905 07:54:23.756184 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.756196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:23.756316 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:23.756640 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:23.756643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:23.756668 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:23.756896 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.756903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.757775 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.757778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:23.758439 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.758449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.759445 (+ 996us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.759460 (+ 15us) client_negotiation.cc:769] Sending connection context
0905 07:54:23.759627 (+ 167us) client_negotiation.cc:240] Negotiation successful
0905 07:54:23.759686 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":108,"threads_started":1}
I20250905 07:54:23.759788 12372 tablet_service.cc:1468] Processing CreateTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 (DEFAULT_TABLE table=test-table [id=f8afaf01bf5642f09e83f64abf150486]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:23.760120 12372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a2fa757274d7ebf2f58742e8789a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:23.760180 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.755749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.4.49.2:36415)
0905 07:54:23.756027 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.756032 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.756256 (+ 224us) server_negotiation.cc:408] Connection header received
0905 07:54:23.756428 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.756432 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.756479 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.756524 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:23.756999 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.757665 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.758585 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.759337 (+ 752us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.759375 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.759952 (+ 577us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.760030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20250905 07:54:23.760579 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.756007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.4.49.1:35795)
0905 07:54:23.756278 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.756282 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.756295 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:23.756756 (+ 461us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.756758 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.756802 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.756867 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:23.757261 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.757896 (+ 635us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.759827 (+ 1931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.760339 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.760378 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.760434 (+ 56us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.760482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20250905 07:54:23.761104 12097 tablet_service.cc:1468] Processing CreateTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 (DEFAULT_TABLE table=test-table [id=f8afaf01bf5642f09e83f64abf150486]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:23.761353 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a2fa757274d7ebf2f58742e8789a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:23.762445 12528 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Bootstrap starting.
I20250905 07:54:23.762789 12233 tablet_service.cc:1468] Processing CreateTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 (DEFAULT_TABLE table=test-table [id=f8afaf01bf5642f09e83f64abf150486]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:23.763067 12233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a2fa757274d7ebf2f58742e8789a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:23.763484 12529 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: Bootstrap starting.
I20250905 07:54:23.763626 12528 tablet_bootstrap.cc:654] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:23.764515 12529 tablet_bootstrap.cc:654] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:23.765420 12532 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Bootstrap starting.
I20250905 07:54:23.765686 12529 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: No bootstrap required, opened a new log
I20250905 07:54:23.765781 12529 ts_tablet_manager.cc:1397] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250905 07:54:23.766271 12529 raft_consensus.cc:357] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.766387 12529 raft_consensus.cc:383] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:23.766420 12529 raft_consensus.cc:738] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec78dd4dd4538b0d5fb795790f03f, State: Initialized, Role: FOLLOWER
I20250905 07:54:23.766456 12528 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: No bootstrap required, opened a new log
I20250905 07:54:23.766546 12528 ts_tablet_manager.cc:1397] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:54:23.766562 12532 tablet_bootstrap.cc:654] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:23.766549 12529 consensus_queue.cc:260] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.766894 12529 ts_tablet_manager.cc:1428] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 07:54:23.767014 12528 raft_consensus.cc:357] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.767108 12528 raft_consensus.cc:383] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:23.767136 12528 raft_consensus.cc:738] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83ef27ae9e547a1afe043b7a0b956ea, State: Initialized, Role: FOLLOWER
I20250905 07:54:23.767252 12528 consensus_queue.cc:260] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.767567 12528 ts_tablet_manager.cc:1428] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:23.768265 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.767829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:59727)
0905 07:54:23.768106 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.768163 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20250905 07:54:23.768865 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.768471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:44257)
0905 07:54:23.768722 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.768782 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20250905 07:54:23.769870 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.769263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:37253)
0905 07:54:23.769409 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.769807 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20250905 07:54:23.770916 12440 heartbeater.cc:499] Master 127.4.49.60:45847 was elected leader, sending a full tablet report...
I20250905 07:54:23.771051 12165 heartbeater.cc:499] Master 127.4.49.60:45847 was elected leader, sending a full tablet report...
I20250905 07:54:23.771449 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.771106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:54397)
0905 07:54:23.771323 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.771372 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20250905 07:54:23.773067 12532 tablet_bootstrap.cc:492] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: No bootstrap required, opened a new log
I20250905 07:54:23.773168 12532 ts_tablet_manager.cc:1397] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250905 07:54:23.773582 12532 raft_consensus.cc:357] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.773674 12532 raft_consensus.cc:383] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:23.773705 12532 raft_consensus.cc:738] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477f5d65c7ca4465ac2287c067ffbfdc, State: Initialized, Role: FOLLOWER
I20250905 07:54:23.773837 12532 consensus_queue.cc:260] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.774161 12532 ts_tablet_manager.cc:1428] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:23.774914 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.774452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:49825)
0905 07:54:23.774769 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.774825 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20250905 07:54:23.775704 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.775349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:55177)
0905 07:54:23.775587 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.775637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20250905 07:54:23.775744 12301 heartbeater.cc:499] Master 127.4.49.60:45847 was elected leader, sending a full tablet report...
W20250905 07:54:23.817550 12303 tablet.cc:2378] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:54:23.866371 12535 raft_consensus.cc:491] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:23.866542 12535 raft_consensus.cc:513] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.867663 12535 leader_election.cc:290] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795), 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
I20250905 07:54:23.870774 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.867927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:35795 (local address 127.4.49.3:43339)
0905 07:54:23.868086 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.868102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:23.868220 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:23.868516 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:23.868520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:23.868545 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:23.868841 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.868850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.869652 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.869656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:23.870302 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.870312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.870441 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.870466 (+ 25us) client_negotiation.cc:769] Sending connection context
0905 07:54:23.870515 (+ 49us) client_negotiation.cc:240] Negotiation successful
0905 07:54:23.870583 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250905 07:54:23.870793 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.867928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:36415 (local address 127.4.49.3:50427)
0905 07:54:23.868436 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.868450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:23.868564 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:23.868816 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:23.868818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:23.868832 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:23.869072 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.869081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.869795 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.869798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:23.870377 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:23.870386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.870508 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.870526 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:54:23.870571 (+ 45us) client_negotiation.cc:240] Negotiation successful
0905 07:54:23.870624 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":192,"thread_start_us":136,"threads_started":1}
I20250905 07:54:23.871254 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.868041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:50427 (local address 127.4.49.2:36415)
0905 07:54:23.868193 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.868197 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.868553 (+ 356us) server_negotiation.cc:408] Connection header received
0905 07:54:23.868673 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.868677 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.868728 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.868803 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:23.869189 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.869687 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.870510 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.870974 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.871016 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.871071 (+ 55us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.871138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20250905 07:54:23.871327 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.868021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:43339 (local address 127.4.49.1:35795)
0905 07:54:23.868165 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.868169 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.868187 (+ 18us) server_negotiation.cc:408] Connection header received
0905 07:54:23.868344 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.868348 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.868404 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.868496 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:23.868970 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.869541 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.870439 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.870942 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.871130 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.871179 (+ 49us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.871232 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250905 07:54:23.871762 12117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5a2fa757274d7ebf2f58742e8789a5" candidate_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" is_pre_election: true
I20250905 07:54:23.871788 12254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5a2fa757274d7ebf2f58742e8789a5" candidate_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" is_pre_election: true
I20250905 07:54:23.871928 12117 raft_consensus.cc:2466] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d83ef27ae9e547a1afe043b7a0b956ea in term 0.
I20250905 07:54:23.871975 12254 raft_consensus.cc:2466] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d83ef27ae9e547a1afe043b7a0b956ea in term 0.
I20250905 07:54:23.872375 12325 leader_election.cc:304] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec78dd4dd4538b0d5fb795790f03f, d83ef27ae9e547a1afe043b7a0b956ea; no voters:
I20250905 07:54:23.872650 12535 raft_consensus.cc:2802] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:23.872706 12535 raft_consensus.cc:491] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:23.872732 12535 raft_consensus.cc:3058] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:23.873663 12535 raft_consensus.cc:513] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.874033 12535 leader_election.cc:290] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795), 477f5d65c7ca4465ac2287c067ffbfdc (127.4.49.2:36415)
I20250905 07:54:23.874430 12254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5a2fa757274d7ebf2f58742e8789a5" candidate_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477f5d65c7ca4465ac2287c067ffbfdc"
I20250905 07:54:23.874430 12117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5a2fa757274d7ebf2f58742e8789a5" candidate_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec78dd4dd4538b0d5fb795790f03f"
I20250905 07:54:23.874562 12117 raft_consensus.cc:3058] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:23.874562 12254 raft_consensus.cc:3058] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:23.875607 12254 raft_consensus.cc:2466] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d83ef27ae9e547a1afe043b7a0b956ea in term 1.
I20250905 07:54:23.875607 12117 raft_consensus.cc:2466] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d83ef27ae9e547a1afe043b7a0b956ea in term 1.
I20250905 07:54:23.875921 12325 leader_election.cc:304] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec78dd4dd4538b0d5fb795790f03f, d83ef27ae9e547a1afe043b7a0b956ea; no voters:
I20250905 07:54:23.876111 12535 raft_consensus.cc:2802] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:23.876339 12535 raft_consensus.cc:695] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 LEADER]: Becoming Leader. State: Replica: d83ef27ae9e547a1afe043b7a0b956ea, State: Running, Role: LEADER
I20250905 07:54:23.876617 12535 consensus_queue.cc:237] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } }
I20250905 07:54:23.877190 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.876843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:59719)
0905 07:54:23.876975 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.877107 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20250905 07:54:23.878973 11965 catalog_manager.cc:5582] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea reported cstate change: term changed from 0 to 1, leader changed from <none> to d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3). New cstate: current_term: 1 leader_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d83ef27ae9e547a1afe043b7a0b956ea" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 37791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 } health_report { overall_health: UNKNOWN } } }
I20250905 07:54:23.914790 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.911718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.4.49.3:37791)
0905 07:54:23.911863 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:23.911868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:23.911898 (+ 30us) server_negotiation.cc:408] Connection header received
0905 07:54:23.911983 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:23.911986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:23.912041 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:23.912160 (+ 119us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:23.912481 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.912983 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:23.913616 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:23.913856 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:23.913919 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:23.914390 (+ 471us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:23.914493 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:23.914584 (+ 91us) server_negotiation.cc:300] Negotiation successful
0905 07:54:23.914642 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:23.917586 12117 raft_consensus.cc:1273] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Refusing update from remote peer d83ef27ae9e547a1afe043b7a0b956ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:23.917757 12254 raft_consensus.cc:1273] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Refusing update from remote peer d83ef27ae9e547a1afe043b7a0b956ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:23.918128 12541 consensus_queue.cc:1035] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec78dd4dd4538b0d5fb795790f03f" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 35795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:23.918265 12535 consensus_queue.cc:1035] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "477f5d65c7ca4465ac2287c067ffbfdc" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:23.919911 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.919625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:49635)
0905 07:54:23.919738 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.919805 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20250905 07:54:23.920850 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.920589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:55503)
0905 07:54:23.920710 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.920771 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20250905 07:54:23.922443 12544 mvcc.cc:204] Tried to move back new op lower bound from 7196913106601721856 to 7196913106441388032. Current Snapshot: MvccSnapshot[applied={T|T < 7196913106601721856}]
I20250905 07:54:23.923210 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.922964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.1:37711)
0905 07:54:23.923089 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.923137 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20250905 07:54:23.923622 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.923140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:46411)
0905 07:54:23.923497 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.923550 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20250905 07:54:23.927047 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.926721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.3:42391)
0905 07:54:23.926911 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.926965 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20250905 07:54:23.928550 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.928243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:43263 (local address 127.4.49.2:47673)
0905 07:54:23.928405 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.928468 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20250905 07:54:23.933740 11964 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:23.933902 11964 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:23.937829 11964 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 51ad02d5fd364fd6a0e80253d6953ed0: Sending DeleteTablet for 3 replicas of tablet dd5a2fa757274d7ebf2f58742e8789a5
I20250905 07:54:23.938441 12372 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:23 UTC) from {username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:47650
I20250905 07:54:23.938627 12097 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:23 UTC) from {username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:39852
I20250905 07:54:23.938915 12554 tablet_replica.cc:331] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: stopping tablet replica
I20250905 07:54:23.938956 12233 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5a2fa757274d7ebf2f58742e8789a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:23 UTC) from {username='oryx', principal='oryx/127.4.49.60@KRBTEST.COM'} at 127.0.0.1:37850
I20250905 07:54:23.939008 12555 tablet_replica.cc:331] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: stopping tablet replica
I20250905 07:54:23.939131 12555 raft_consensus.cc:2241] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:23.939160 12554 raft_consensus.cc:2241] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:23.939265 12555 raft_consensus.cc:2270] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:23.939591 12554 raft_consensus.cc:2270] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:23.940030 12555 ts_tablet_manager.cc:1905] T dd5a2fa757274d7ebf2f58742e8789a5 P 1cfec78dd4dd4538b0d5fb795790f03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:23.940165 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12034
I20250905 07:54:23.940436 12554 ts_tablet_manager.cc:1905] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:23.943245 12554 ts_tablet_manager.cc:1918] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:23.943404 12554 log.cc:1199] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/dd5a2fa757274d7ebf2f58742e8789a5
I20250905 07:54:23.943727 12554 ts_tablet_manager.cc:1939] T dd5a2fa757274d7ebf2f58742e8789a5 P d83ef27ae9e547a1afe043b7a0b956ea: Deleting consensus metadata
I20250905 07:54:23.944975 11952 catalog_manager.cc:4928] TS d83ef27ae9e547a1afe043b7a0b956ea (127.4.49.3:37791): tablet dd5a2fa757274d7ebf2f58742e8789a5 (table test-table [id=f8afaf01bf5642f09e83f64abf150486]) successfully deleted
I20250905 07:54:23.948385 12556 tablet_replica.cc:331] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: stopping tablet replica
I20250905 07:54:23.948530 12556 raft_consensus.cc:2241] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:23.948691 12556 raft_consensus.cc:2270] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:23.949427 12556 ts_tablet_manager.cc:1905] T dd5a2fa757274d7ebf2f58742e8789a5 P 477f5d65c7ca4465ac2287c067ffbfdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:23.950824 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12171
I20250905 07:54:23.951704 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.950947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:35795 (local address 127.0.0.1:39862)
0905 07:54:23.951558 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.951613 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:35795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524}
W20250905 07:54:23.951867 11950 catalog_manager.cc:4671] TS 1cfec78dd4dd4538b0d5fb795790f03f (127.4.49.1:35795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd5a2fa757274d7ebf2f58742e8789a5: Network error: Client connection negotiation failed: client connection to 127.4.49.1:35795: connect: Connection refused (error 111)
I20250905 07:54:23.960075 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12307
I20250905 07:54:23.961828 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:23.960201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:36415 (local address 127.0.0.1:37864)
0905 07:54:23.961680 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:23.961740 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.2:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1377}
I20250905 07:54:23.965611 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11632
I20250905 07:54:23.972798 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 11747
2025-09-05T07:54:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8290 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:24 dist-test-slave-mnfb krb5kdc[12559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:24 dist-test-slave-mnfb krb5kdc[12559](info): set up 2 sockets
Sep 05 07:54:24 dist-test-slave-mnfb krb5kdc[12559](info): commencing operation
krb5kdc: starting...
W20250905 07:54:25.807902 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:25 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:25.942314 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:42367
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:26.050894 12575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:26.051168 12575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:26.051246 12575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:26.054750 12575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:26.054849 12575 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250905 07:54:26.054873 12575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:26.054900 12575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:26.054924 12575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:26.054948 12575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:26.059733 12575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:42367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:42367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:26.060945 12575 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:26.061894 12575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:26.067907 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.067960 12581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.067984 12580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:26.068312 12575 server_base.cc:1047] running on GCE node
I20250905 07:54:26.068881 12575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:26.069909 12575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:26.071097 12575 hybrid_clock.cc:648] HybridClock initialized: now 1757058866071086 us; error 43 us; skew 500 ppm
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:26.074177 12575 init.cc:377] Logged in from keytab as oryx/127.4.49.62@KRBTEST.COM (short username oryx)
I20250905 07:54:26.075388 12575 webserver.cc:480] Webserver started at http://127.4.49.62:35105/ using document root <none> and password file <none>
I20250905 07:54:26.075985 12575 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:26.076045 12575 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:26.076265 12575 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:26.078228 12575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "ed0b16e84e324b309baa85cd3d457dd6"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "4e6156129aa103090aa38f1bc0a62f86"
server_key_iv: "bb8588ee258448f42246b08e2a8a19b1"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.078742 12575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "ed0b16e84e324b309baa85cd3d457dd6"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "4e6156129aa103090aa38f1bc0a62f86"
server_key_iv: "bb8588ee258448f42246b08e2a8a19b1"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.082407 12575 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250905 07:54:26.085045 12590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.086196 12575 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:26.086351 12575 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "ed0b16e84e324b309baa85cd3d457dd6"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "4e6156129aa103090aa38f1bc0a62f86"
server_key_iv: "bb8588ee258448f42246b08e2a8a19b1"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.086468 12575 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:26.100564 12575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:26.103425 12575 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:26.103623 12575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:26.111848 12575 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:42367
I20250905 07:54:26.111876 12653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:42367 every 8 connection(s)
I20250905 07:54:26.112999 12575 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:26.116084 12654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:26.118299 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12575
I20250905 07:54:26.118404 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:26.118623 4292 external_mini_cluster.cc:1468] Setting key 644b7c38b08b29232089a531ea8c05ac
I20250905 07:54:26.122684 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Bootstrap starting.
I20250905 07:54:26.125162 12654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:26.125948 12654 log.cc:826] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Log is configured to *not* fsync() on all Append() calls
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:26.128302 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: No bootstrap required, opened a new log
I20250905 07:54:26.131714 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.119954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.4.49.62:42367)
0905 07:54:26.120480 (+ 526us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.120491 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.120532 (+ 41us) server_negotiation.cc:408] Connection header received
0905 07:54:26.121115 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.121135 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.121458 (+ 323us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.121799 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.122734 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.123518 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.124213 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.124504 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.126727 (+ 2223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.126748 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.126762 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.126807 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.128853 (+ 2046us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.129276 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.129282 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.129288 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.129407 (+ 119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.129612 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.129615 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.129617 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.130073 (+ 456us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.130267 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.130566 (+ 299us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.130866 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":177,"threads_started":1}
I20250905 07:54:26.132027 12654 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } }
I20250905 07:54:26.132293 12654 raft_consensus.cc:383] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:26.132359 12654 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0b16e84e324b309baa85cd3d457dd6, State: Initialized, Role: FOLLOWER
I20250905 07:54:26.132846 12654 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } }
I20250905 07:54:26.132999 12654 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:26.133062 12654 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:26.133176 12654 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:26.134382 12654 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } }
I20250905 07:54:26.134759 12654 leader_election.cc:304] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed0b16e84e324b309baa85cd3d457dd6; no voters:
I20250905 07:54:26.135078 12654 leader_election.cc:290] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:26.135291 12659 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:26.135550 12659 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [term 1 LEADER]: Becoming Leader. State: Replica: ed0b16e84e324b309baa85cd3d457dd6, State: Running, Role: LEADER
I20250905 07:54:26.135854 12659 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } }
I20250905 07:54:26.136271 12654 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:26.137449 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed0b16e84e324b309baa85cd3d457dd6. Latest consensus state: current_term: 1 leader_uuid: "ed0b16e84e324b309baa85cd3d457dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } } }
I20250905 07:54:26.137589 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:26.138028 12660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed0b16e84e324b309baa85cd3d457dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0b16e84e324b309baa85cd3d457dd6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 42367 } } }
I20250905 07:54:26.138114 12668 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:26.138134 12660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:26.141793 12668 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:26.148114 12668 catalog_manager.cc:1349] Generated new cluster ID: 3f35ea2fdb4c47d0ae96507f8a854cc6
I20250905 07:54:26.148211 12668 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:26.159101 12668 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:26.160480 12668 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:26.171226 12668 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Generated new TSK 0
I20250905 07:54:26.172310 12668 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:26.236537 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42367
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:26.359265 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:26.359522 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:26.359586 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:26.363143 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:26.363240 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:26.363298 12682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:54:26.363350 12682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:26.368036 12682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:26.369298 12682 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:26.370230 12682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:26.377473 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.377487 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.377473 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:26.377557 12682 server_base.cc:1047] running on GCE node
I20250905 07:54:26.378115 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:26.378739 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:26.379935 12682 hybrid_clock.cc:648] HybridClock initialized: now 1757058866379917 us; error 34 us; skew 500 ppm
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:26.383060 12682 init.cc:377] Logged in from keytab as oryx/127.4.49.1@KRBTEST.COM (short username oryx)
I20250905 07:54:26.384300 12682 webserver.cc:480] Webserver started at http://127.4.49.1:33027/ using document root <none> and password file <none>
I20250905 07:54:26.384905 12682 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:26.384958 12682 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:26.385162 12682 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:26.386998 12682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "ae18241e0a6c4438889bb2171b1985e2"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "8e8f8d981f8df2aa9e5451f14528b5ab"
server_key_iv: "c9619b78c39e40b1f8ebb4f98eeb42ba"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.387480 12682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "ae18241e0a6c4438889bb2171b1985e2"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "8e8f8d981f8df2aa9e5451f14528b5ab"
server_key_iv: "c9619b78c39e40b1f8ebb4f98eeb42ba"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.391187 12682 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250905 07:54:26.393695 12697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.394850 12682 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:54:26.394977 12682 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "ae18241e0a6c4438889bb2171b1985e2"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "8e8f8d981f8df2aa9e5451f14528b5ab"
server_key_iv: "c9619b78c39e40b1f8ebb4f98eeb42ba"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.395092 12682 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:26.408944 12682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:26.416126 12682 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:26.416324 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:26.417933 12682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:26.417995 12682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.418040 12682 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:26.418056 12682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:26.429461 12682 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:33047
I20250905 07:54:26.429481 12813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:33047 every 8 connection(s)
I20250905 07:54:26.430701 12682 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:26.432031 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.418750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:59885 (local address 127.4.49.62:42367)
0905 07:54:26.418918 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.418923 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.419667 (+ 744us) server_negotiation.cc:408] Connection header received
0905 07:54:26.420728 (+ 1061us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.420733 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.420797 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.420905 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.422550 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.423367 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.424238 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.424464 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.427507 (+ 3043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.427533 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.427535 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.427573 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.429511 (+ 1938us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.430168 (+ 657us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.430172 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.430174 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.430231 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.430572 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.430574 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.430576 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.430765 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.430858 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.431773 (+ 915us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.431899 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250905 07:54:26.432446 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.419016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.1:59885)
0905 07:54:26.419509 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.419555 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.420471 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.421188 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.421199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.421676 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.422341 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.422361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.423531 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.423535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.424027 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.424038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.424319 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.425245 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.425292 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.427270 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.429700 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.429706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.429719 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.430044 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.430333 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.430336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.430338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.430463 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.430911 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.430917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.431185 (+ 268us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.431407 (+ 222us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.431720 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20250905 07:54:26.433683 12814 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42367
I20250905 07:54:26.433733 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12682
I20250905 07:54:26.433836 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:26.433871 12814 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:26.434115 4292 external_mini_cluster.cc:1468] Setting key a4a5a7b235a7d880b47e7bdb6f029f81
I20250905 07:54:26.434451 12814 heartbeater.cc:507] Master 127.4.49.62:42367 requested a full tablet report, sending...
I20250905 07:54:26.436117 12607 ts_manager.cc:194] Registered new tserver with Master: ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047)
I20250905 07:54:26.437943 12607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:59885
I20250905 07:54:26.444970 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.436220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.1:46941)
0905 07:54:26.436566 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.436581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.436695 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.437029 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.437032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.437247 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.437554 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.437563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.438391 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.438395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.440391 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.440404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.440546 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.441173 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.441188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.441518 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.443562 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.443566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.443567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.443761 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.444022 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.444024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.444026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.444116 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.444528 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.444530 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.444610 (+ 80us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.444691 (+ 81us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.444814 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:54:26.445060 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.436288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:46941 (local address 127.4.49.62:42367)
0905 07:54:26.436406 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.436411 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.436653 (+ 242us) server_negotiation.cc:408] Connection header received
0905 07:54:26.436781 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.436784 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.436832 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.436908 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.437742 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.438288 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.440559 (+ 2271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.440724 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.441670 (+ 946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.441697 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.441701 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.441735 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.443415 (+ 1680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.443867 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.443871 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.443873 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.443936 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.444198 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.444202 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.444204 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.444439 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.444551 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.444766 (+ 215us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.444898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:26.481027 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42367
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:26.592825 12825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:26.593073 12825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:26.593137 12825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:26.596657 12825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:26.596740 12825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:26.596796 12825 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:54:26.596838 12825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:26.601627 12825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:26.602794 12825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:26.603837 12825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:26.611037 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.611043 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.611068 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:26.611337 12825 server_base.cc:1047] running on GCE node
I20250905 07:54:26.611754 12825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:26.612353 12825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:26.613514 12825 hybrid_clock.cc:648] HybridClock initialized: now 1757058866613488 us; error 38 us; skew 500 ppm
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:26.616655 12825 init.cc:377] Logged in from keytab as oryx/127.4.49.2@KRBTEST.COM (short username oryx)
I20250905 07:54:26.617985 12825 webserver.cc:480] Webserver started at http://127.4.49.2:33131/ using document root <none> and password file <none>
I20250905 07:54:26.618563 12825 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:26.618616 12825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:26.618821 12825 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:26.620849 12825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "d7c51279dee45c2f9459c0c1356192aa"
server_key_iv: "beef3a69323cb1aac760d25776219097"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.621351 12825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "d7c51279dee45c2f9459c0c1356192aa"
server_key_iv: "beef3a69323cb1aac760d25776219097"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.625058 12825 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:54:26.627722 12840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.629052 12825 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:54:26.629189 12825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "d7c51279dee45c2f9459c0c1356192aa"
server_key_iv: "beef3a69323cb1aac760d25776219097"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.629292 12825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:26.652953 12825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:26.656294 12825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:26.656483 12825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:26.658020 12825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:26.658087 12825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.658140 12825 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:26.658169 12825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:26.669095 12825 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:44591
I20250905 07:54:26.669113 12956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:44591 every 8 connection(s)
I20250905 07:54:26.670336 12825 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:26.672236 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.659140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.2:42395)
0905 07:54:26.659607 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.659646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.660486 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.661063 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.661076 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.661695 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.662603 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.662621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.663840 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.663846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.664500 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.664511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.664768 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.665482 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.665526 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.667459 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.669625 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.669631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.669641 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.669896 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.670161 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.670163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.670165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.670255 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.670686 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.670694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.671005 (+ 311us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.671227 (+ 222us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.671490 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
I20250905 07:54:26.672876 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.658776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:42395 (local address 127.4.49.62:42367)
0905 07:54:26.658907 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.658912 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.659755 (+ 843us) server_negotiation.cc:408] Connection header received
0905 07:54:26.660718 (+ 963us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.660722 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.660788 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.660862 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.662838 (+ 1976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.663670 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.664648 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.664870 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.667671 (+ 2801us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.667701 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.667706 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.667748 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.669487 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.670008 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.670012 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.670014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.670077 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.670341 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.670348 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.670350 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.670580 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.670694 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.672629 (+ 1935us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.672758 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:54:26.673871 12957 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42367
I20250905 07:54:26.674052 12957 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:26.674762 12957 heartbeater.cc:507] Master 127.4.49.62:42367 requested a full tablet report, sending...
I20250905 07:54:26.675998 12606 ts_manager.cc:194] Registered new tserver with Master: 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:26.676684 12606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:42395
I20250905 07:54:26.677371 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12825
I20250905 07:54:26.677527 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:26.677793 4292 external_mini_cluster.cc:1468] Setting key fdef3853f4ce7605be73eaeb1f4bb880
I20250905 07:54:26.682797 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.676159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:46137 (local address 127.4.49.62:42367)
0905 07:54:26.676292 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.676296 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.676476 (+ 180us) server_negotiation.cc:408] Connection header received
0905 07:54:26.676597 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.676600 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.676644 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.676711 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.677397 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.677993 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.678736 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.678920 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.679755 (+ 835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.679770 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.679772 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.679795 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.681364 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.681837 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.681840 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.681841 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.681885 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.682086 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.682088 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.682089 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.682256 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.682326 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.682566 (+ 240us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.682669 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250905 07:54:26.682885 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.676119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.2:46137)
0905 07:54:26.676398 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.676410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.676511 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.676755 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.676758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.676963 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.677243 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.677252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.678109 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.678112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.678600 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.678609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.678731 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.679328 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.679341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.679625 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.681529 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.681533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.681535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.681747 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.681960 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.681963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.681965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.682021 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.682329 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.682332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.682427 (+ 95us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.682530 (+ 103us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.682684 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab.
I20250905 07:54:26.720821 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:42367
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:26.830518 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:26.830771 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:26.830837 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:26.834302 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:26.834384 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:26.834450 12968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250905 07:54:26.834522 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:26.839218 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.49.62:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:26.840368 12968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:26.841228 12968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:26.847929 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.847929 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:26.847929 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:26.848192 12968 server_base.cc:1047] running on GCE node
I20250905 07:54:26.848644 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:26.849222 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:26.850373 12968 hybrid_clock.cc:648] HybridClock initialized: now 1757058866850354 us; error 35 us; skew 500 ppm
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:26.853366 12968 init.cc:377] Logged in from keytab as oryx/127.4.49.3@KRBTEST.COM (short username oryx)
I20250905 07:54:26.854547 12968 webserver.cc:480] Webserver started at http://127.4.49.3:42767/ using document root <none> and password file <none>
I20250905 07:54:26.855100 12968 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:26.855149 12968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:26.855312 12968 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:26.857201 12968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "4cad7908d7d14f30ad901ea6b91babec"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "fcb27efae08ca05a0b2c64362c63de05"
server_key_iv: "8b3aa4de30ce289a35255fa19413c959"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.857717 12968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "4cad7908d7d14f30ad901ea6b91babec"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "fcb27efae08ca05a0b2c64362c63de05"
server_key_iv: "8b3aa4de30ce289a35255fa19413c959"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.861371 12968 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:26.863792 12983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.864979 12968 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:26.865103 12968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "4cad7908d7d14f30ad901ea6b91babec"
format_stamp: "Formatted at 2025-09-05 07:54:26 on dist-test-slave-mnfb"
server_key: "fcb27efae08ca05a0b2c64362c63de05"
server_key_iv: "8b3aa4de30ce289a35255fa19413c959"
server_key_version: "encryptionkey@0"
I20250905 07:54:26.865199 12968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:26.886492 12968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:26.889602 12968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:26.889778 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:26.891314 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:26.891377 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:26.891426 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:26.891454 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:26.901976 12968 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:34645
I20250905 07:54:26.902016 13099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:34645 every 8 connection(s)
I20250905 07:54:26.902985 12968 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:26.904218 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.892110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:36207 (local address 127.4.49.62:42367)
0905 07:54:26.892261 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.892266 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.893039 (+ 773us) server_negotiation.cc:408] Connection header received
0905 07:54:26.894070 (+ 1031us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.894075 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.894137 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.894194 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.896216 (+ 2022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.896885 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.897600 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.897782 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.900539 (+ 2757us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.900570 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.900574 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.900610 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.902133 (+ 1523us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.902650 (+ 517us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.902653 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.902654 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.902710 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.902980 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.902983 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.902986 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.903187 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.903277 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.903883 (+ 606us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.904022 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250905 07:54:26.904843 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.892434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.3:36207)
0905 07:54:26.892875 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.892919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.893812 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.894644 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.894658 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.895181 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.896012 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.896031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.897024 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.897028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.897473 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.897481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.897710 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.898443 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.898499 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.900344 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.902260 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.902267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.902280 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.902534 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.902789 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.902792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.902794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.902886 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.903288 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.903296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.903589 (+ 293us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.903815 (+ 226us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.904133 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20250905 07:54:26.905957 13100 heartbeater.cc:344] Connected to a master server at 127.4.49.62:42367
I20250905 07:54:26.906126 13100 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:26.906730 13100 heartbeater.cc:507] Master 127.4.49.62:42367 requested a full tablet report, sending...
I20250905 07:54:26.906874 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 12968
I20250905 07:54:26.906977 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:26.907262 4292 external_mini_cluster.cc:1468] Setting key d69854d0caa68a7021064e1c0649f42f
I20250905 07:54:26.907897 12607 ts_manager.cc:194] Registered new tserver with Master: 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645)
I20250905 07:54:26.908691 12607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:36207
I20250905 07:54:26.909452 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:54:26.915081 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.908354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.4.49.3:33587)
0905 07:54:26.908647 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.908662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.908821 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.909136 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.909139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.909357 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:26.909655 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.909663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.910348 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.910352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.910752 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.910759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.910866 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.911471 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:26.911491 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:26.911906 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:26.913680 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.913684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.913687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.913915 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.914165 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:26.914168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:26.914169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:26.914222 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:26.914618 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:26.914621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:26.914721 (+ 100us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.914801 (+ 80us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.914925 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
I20250905 07:54:26.915148 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.908234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:33587 (local address 127.4.49.62:42367)
0905 07:54:26.908352 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.908357 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.908744 (+ 387us) server_negotiation.cc:408] Connection header received
0905 07:54:26.908911 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.908914 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.908961 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.909023 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.909767 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.910248 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.910857 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.911040 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.912093 (+ 1053us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.912114 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.912117 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.912144 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.913552 (+ 1408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.914016 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.914019 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.914020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.914075 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.914303 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.914306 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.914307 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.914536 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.914635 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.914868 (+ 233us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.914992 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
W20250905 07:54:26.932610 12657 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:26.926958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.4.49.62:42367)
0905 07:54:26.927115 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.927118 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.927131 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:26.927172 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.927175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.927220 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.927293 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.928222 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.928744 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.929406 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.929605 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.932478 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44522: BlockingRecv error: recv got EOF from 127.0.0.1:44522 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 05 07:54:26 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:26.941184 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.933967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.4.49.62:42367)
0905 07:54:26.934090 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.934094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.934155 (+ 61us) server_negotiation.cc:408] Connection header received
0905 07:54:26.934296 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.934300 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.934344 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.934411 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:26.935200 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.935682 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.936330 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.936515 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.938562 (+ 2047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:26.938588 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:26.938592 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:26.938625 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:26.939738 (+ 1113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.940128 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.940132 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.940134 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.940184 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:26.940381 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:26.940384 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:26.940386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:26.940549 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:26.940767 (+ 218us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.940914 (+ 147us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.941028 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250905 07:54:26.943953 12606 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:26.946542 12606 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:26.960335 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.956310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:44591 (local address 127.0.0.1:58694)
0905 07:54:26.956820 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.956831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.956910 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.957651 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.957655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.957670 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:26.957857 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.957863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.958836 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.958840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.959693 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.959702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.959809 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.959888 (+ 79us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.959983 (+ 95us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.960120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"spinlock_wait_cycles":6272,"thread_start_us":101,"threads_started":1}
I20250905 07:54:26.960335 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.955961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:33047 (local address 127.0.0.1:47612)
0905 07:54:26.956787 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.956800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.956910 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.957442 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.957445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.957465 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:26.957700 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.957708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.959015 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.959018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.959770 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.959780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.959876 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.959896 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.959984 (+ 88us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.960160 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"spinlock_wait_cycles":6016,"thread_start_us":116,"threads_started":1}
I20250905 07:54:26.960894 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.955199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:34645 (local address 127.0.0.1:35922)
0905 07:54:26.956226 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:26.956265 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:26.956417 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:26.957192 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:26.957198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:26.957222 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:26.957555 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.957565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.958975 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.958979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:26.960123 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:26.960131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.960220 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.960237 (+ 17us) client_negotiation.cc:769] Sending connection context
0905 07:54:26.960280 (+ 43us) client_negotiation.cc:240] Negotiation successful
0905 07:54:26.960335 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":105,"threads_started":1}
I20250905 07:54:26.961167 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.956473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.4.49.2:44591)
0905 07:54:26.957119 (+ 646us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.957123 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.957143 (+ 20us) server_negotiation.cc:408] Connection header received
0905 07:54:26.957211 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.957215 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.957342 (+ 127us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.957486 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:26.957964 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.958708 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.960185 (+ 1477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.960728 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.960863 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.960933 (+ 70us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.961027 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":83,"threads_started":1}
I20250905 07:54:26.961505 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.956138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.4.49.1:33047)
0905 07:54:26.956557 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.956563 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.957033 (+ 470us) server_negotiation.cc:408] Connection header received
0905 07:54:26.957121 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.957126 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.957281 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.957432 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:26.957826 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.958892 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.960411 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.961137 (+ 726us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.961248 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.961311 (+ 63us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.961393 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":92,"threads_started":1}
I20250905 07:54:26.961828 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:26.955308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.4.49.3:34645)
0905 07:54:26.955671 (+ 363us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:26.955677 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:26.956569 (+ 892us) server_negotiation.cc:408] Connection header received
0905 07:54:26.956651 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:26.956656 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:26.956845 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:26.956982 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:26.957670 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.958854 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:26.960434 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:26.961460 (+ 1026us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:26.961553 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:26.961613 (+ 60us) server_negotiation.cc:300] Negotiation successful
0905 07:54:26.961695 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20250905 07:54:26.964179 13033 tablet_service.cc:1468] Processing CreateTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (DEFAULT_TABLE table=test-table [id=c6b0bdfa236c4e9d9eac06773f972804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:26.964195 12747 tablet_service.cc:1468] Processing CreateTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (DEFAULT_TABLE table=test-table [id=c6b0bdfa236c4e9d9eac06773f972804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:26.964200 12890 tablet_service.cc:1468] Processing CreateTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (DEFAULT_TABLE table=test-table [id=c6b0bdfa236c4e9d9eac06773f972804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:26.965201 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:26.965204 12890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:26.965204 12747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:26.971431 13129 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Bootstrap starting.
I20250905 07:54:26.972249 13130 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Bootstrap starting.
I20250905 07:54:26.973774 13129 tablet_bootstrap.cc:654] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:26.974238 13130 tablet_bootstrap.cc:654] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:26.974880 13129 log.cc:826] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:26.974890 13130 log.cc:826] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:26.975622 13131 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Bootstrap starting.
I20250905 07:54:26.976718 13129 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: No bootstrap required, opened a new log
I20250905 07:54:26.976933 13129 ts_tablet_manager.cc:1397] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250905 07:54:26.977255 13130 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: No bootstrap required, opened a new log
I20250905 07:54:26.977728 13130 ts_tablet_manager.cc:1397] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250905 07:54:26.977999 13131 tablet_bootstrap.cc:654] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:26.979286 13131 log.cc:826] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:26.979730 13129 raft_consensus.cc:357] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.979984 13129 raft_consensus.cc:383] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:26.980047 13129 raft_consensus.cc:738] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Initialized, Role: FOLLOWER
I20250905 07:54:26.980530 13129 consensus_queue.cc:260] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.980798 13130 raft_consensus.cc:357] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.981021 13130 raft_consensus.cc:383] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:26.981079 13130 raft_consensus.cc:738] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Initialized, Role: FOLLOWER
I20250905 07:54:26.981359 13100 heartbeater.cc:499] Master 127.4.49.62:42367 was elected leader, sending a full tablet report...
I20250905 07:54:26.981460 13129 ts_tablet_manager.cc:1428] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250905 07:54:26.981508 13130 consensus_queue.cc:260] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.982168 12814 heartbeater.cc:499] Master 127.4.49.62:42367 was elected leader, sending a full tablet report...
I20250905 07:54:26.982385 13130 ts_tablet_manager.cc:1428] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:54:26.982534 13131 tablet_bootstrap.cc:492] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: No bootstrap required, opened a new log
I20250905 07:54:26.982746 13131 ts_tablet_manager.cc:1397] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250905 07:54:26.985929 13131 raft_consensus.cc:357] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.986227 13131 raft_consensus.cc:383] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:26.986291 13131 raft_consensus.cc:738] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Initialized, Role: FOLLOWER
I20250905 07:54:26.986786 13131 consensus_queue.cc:260] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:26.987443 12957 heartbeater.cc:499] Master 127.4.49.62:42367 was elected leader, sending a full tablet report...
I20250905 07:54:26.987634 13131 ts_tablet_manager.cc:1428] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250905 07:54:27.022770 13135 raft_consensus.cc:491] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:27.022995 13135 raft_consensus.cc:513] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:27.024284 13135 leader_election.cc:290] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.027540 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.024487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:33047 (local address 127.4.49.3:38277)
0905 07:54:27.024622 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.024643 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.024732 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.025032 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.025036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.025057 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.025306 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.025314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.026263 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.026266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.027106 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.027117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.027237 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.027261 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.027313 (+ 52us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.027385 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20250905 07:54:27.027585 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.024742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:44591 (local address 127.4.49.3:55487)
0905 07:54:27.025036 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.025048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.025128 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.025331 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.025334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.025352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.025589 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.025594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.026477 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.026481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.027279 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.027287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.027376 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.027388 (+ 12us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.027428 (+ 40us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.027472 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":120,"threads_started":1}
I20250905 07:54:27.027989 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.024598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:38277 (local address 127.4.49.1:33047)
0905 07:54:27.024739 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.024743 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.024759 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:27.024817 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.024820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.024872 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.024953 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.025422 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.026123 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.027242 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.027718 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.027761 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.027814 (+ 53us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.027874 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250905 07:54:27.028122 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.024873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:55487 (local address 127.4.49.2:44591)
0905 07:54:27.024998 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.025002 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.025110 (+ 108us) server_negotiation.cc:408] Connection header received
0905 07:54:27.025204 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.025206 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.025246 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.025304 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.025694 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.026363 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.027383 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.027850 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.027884 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.027930 (+ 46us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.027987 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:27.028668 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbaea3fecb7d49d2aabd92b9b80e4ff1" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2" is_pre_election: true
I20250905 07:54:27.028668 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbaea3fecb7d49d2aabd92b9b80e4ff1" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" is_pre_election: true
I20250905 07:54:27.028975 12767 raft_consensus.cc:2466] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.028975 12910 raft_consensus.cc:2466] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.029484 12987 leader_election.cc:304] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad7908d7d14f30ad901ea6b91babec, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.029731 13135 raft_consensus.cc:2802] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:27.029811 13135 raft_consensus.cc:491] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:27.029841 13135 raft_consensus.cc:3058] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.030812 13135 raft_consensus.cc:513] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:27.031176 13135 leader_election.cc:290] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Requested vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.031559 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbaea3fecb7d49d2aabd92b9b80e4ff1" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2"
I20250905 07:54:27.031639 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbaea3fecb7d49d2aabd92b9b80e4ff1" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
I20250905 07:54:27.031698 12767 raft_consensus.cc:3058] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.031778 12910 raft_consensus.cc:3058] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.033100 12910 raft_consensus.cc:2466] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.033371 12767 raft_consensus.cc:2466] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.033490 12987 leader_election.cc:304] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, 4cad7908d7d14f30ad901ea6b91babec; no voters:
I20250905 07:54:27.033694 13135 raft_consensus.cc:2802] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:27.033943 13135 raft_consensus.cc:695] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Becoming Leader. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Running, Role: LEADER
I20250905 07:54:27.034245 13135 consensus_queue.cc:237] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } }
I20250905 07:54:27.037459 12607 catalog_manager.cc:5582] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3). New cstate: current_term: 1 leader_uuid: "4cad7908d7d14f30ad901ea6b91babec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } health_report { overall_health: UNKNOWN } } }
I20250905 07:54:27.066104 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.062990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.4.49.3:34645)
0905 07:54:27.063135 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.063140 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.063156 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:27.063244 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.063247 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.063304 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.063386 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.063741 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.064282 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.064973 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.065161 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.065219 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.065709 (+ 490us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.065814 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.065901 (+ 87us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.065963 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:54:27.071902 12767 raft_consensus.cc:1273] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.071902 12910 raft_consensus.cc:1273] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.072571 13139 consensus_queue.cc:1035] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.072741 13135 consensus_queue.cc:1035] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.089605 12606 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.089772 12606 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.092337 12606 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Sending DeleteTablet for 3 replicas of tablet bbaea3fecb7d49d2aabd92b9b80e4ff1
I20250905 07:54:27.093148 13033 tablet_service.cc:1515] Processing DeleteTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:35922
I20250905 07:54:27.093479 12890 tablet_service.cc:1515] Processing DeleteTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:58694
I20250905 07:54:27.093724 12747 tablet_service.cc:1515] Processing DeleteTablet for tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:47612
I20250905 07:54:27.093822 12606 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:54:27.094024 13153 tablet_replica.cc:331] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: stopping tablet replica
W20250905 07:54:27.094286 12606 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:27.094308 13153 raft_consensus.cc:2241] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.094558 13152 tablet_replica.cc:331] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: stopping tablet replica
I20250905 07:54:27.094689 13153 raft_consensus.cc:2270] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.094921 13152 raft_consensus.cc:2241] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:27.095386 13152 raft_consensus.cc:2270] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.096628 13153 ts_tablet_manager.cc:1905] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.097085 13152 ts_tablet_manager.cc:1905] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.099421 13153 ts_tablet_manager.cc:1918] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.099524 13153 log.cc:1199] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/bbaea3fecb7d49d2aabd92b9b80e4ff1
I20250905 07:54:27.099781 13154 tablet_replica.cc:331] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: stopping tablet replica
I20250905 07:54:27.099813 13153 ts_tablet_manager.cc:1939] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting consensus metadata
I20250905 07:54:27.100203 12890 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf (DEFAULT_TABLE table=test-table [id=9abb8ff828da4517940e29bb1981a955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.100204 13033 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf (DEFAULT_TABLE table=test-table [id=9abb8ff828da4517940e29bb1981a955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.100450 12890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba63065fcf6407a83a8a9ac898e0daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.100486 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba63065fcf6407a83a8a9ac898e0daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.101333 12594 catalog_manager.cc:4928] TS 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591): tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (table test-table [id=c6b0bdfa236c4e9d9eac06773f972804]) successfully deleted
I20250905 07:54:27.102654 13129 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Bootstrap starting.
I20250905 07:54:27.102821 13131 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Bootstrap starting.
I20250905 07:54:27.103796 13129 tablet_bootstrap.cc:654] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.103796 13131 tablet_bootstrap.cc:654] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.102913 12747 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf (DEFAULT_TABLE table=test-table [id=9abb8ff828da4517940e29bb1981a955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.104163 12747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba63065fcf6407a83a8a9ac898e0daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.105664 13131 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: No bootstrap required, opened a new log
I20250905 07:54:27.105746 13131 ts_tablet_manager.cc:1397] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:27.105886 13129 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: No bootstrap required, opened a new log
I20250905 07:54:27.105955 13129 ts_tablet_manager.cc:1397] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:27.106236 13131 raft_consensus.cc:357] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.106336 13131 raft_consensus.cc:383] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.106369 13131 raft_consensus.cc:738] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.106359 13129 raft_consensus.cc:357] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.106442 13130 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Bootstrap starting.
I20250905 07:54:27.106494 13129 raft_consensus.cc:383] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.106526 13129 raft_consensus.cc:738] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.106493 13131 consensus_queue.cc:260] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.106637 13129 consensus_queue.cc:260] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.106760 13131 ts_tablet_manager.cc:1428] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.106856 13129 ts_tablet_manager.cc:1428] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.107484 13130 tablet_bootstrap.cc:654] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.108853 13154 raft_consensus.cc:2241] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.109618 13130 tablet_bootstrap.cc:492] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: No bootstrap required, opened a new log
I20250905 07:54:27.109704 13130 ts_tablet_manager.cc:1397] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250905 07:54:27.109833 13154 raft_consensus.cc:2270] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.110164 13130 raft_consensus.cc:357] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.110265 13130 raft_consensus.cc:383] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.110293 13130 raft_consensus.cc:738] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.110406 13130 consensus_queue.cc:260] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.110668 13130 ts_tablet_manager.cc:1428] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 07:54:27.109174 13152 ts_tablet_manager.cc:1918] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.111238 13152 log.cc:1199] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/bbaea3fecb7d49d2aabd92b9b80e4ff1
I20250905 07:54:27.111613 13152 ts_tablet_manager.cc:1939] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P 4cad7908d7d14f30ad901ea6b91babec: Deleting consensus metadata
I20250905 07:54:27.112325 13154 ts_tablet_manager.cc:1905] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.112872 12594 catalog_manager.cc:4928] TS 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645): tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (table test-table [id=c6b0bdfa236c4e9d9eac06773f972804]) successfully deleted
I20250905 07:54:27.115617 13154 ts_tablet_manager.cc:1918] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.115717 13154 log.cc:1199] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/bbaea3fecb7d49d2aabd92b9b80e4ff1
I20250905 07:54:27.116024 13154 ts_tablet_manager.cc:1939] T bbaea3fecb7d49d2aabd92b9b80e4ff1 P ae18241e0a6c4438889bb2171b1985e2: Deleting consensus metadata
I20250905 07:54:27.117687 12594 catalog_manager.cc:4928] TS ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047): tablet bbaea3fecb7d49d2aabd92b9b80e4ff1 (table test-table [id=c6b0bdfa236c4e9d9eac06773f972804]) successfully deleted
W20250905 07:54:27.153899 13101 tablet.cc:2378] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:54:27.171669 12958 tablet.cc:2378] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:54:27.182024 12815 tablet.cc:2378] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:54:27.279350 13147 raft_consensus.cc:491] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:27.279527 13147 raft_consensus.cc:513] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.279973 13147 leader_election.cc:290] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.280575 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba63065fcf6407a83a8a9ac898e0daf" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" is_pre_election: true
I20250905 07:54:27.280575 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba63065fcf6407a83a8a9ac898e0daf" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2" is_pre_election: true
I20250905 07:54:27.280742 12767 raft_consensus.cc:2466] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.280742 12910 raft_consensus.cc:2466] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.281059 12987 leader_election.cc:304] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, 4cad7908d7d14f30ad901ea6b91babec; no voters:
I20250905 07:54:27.281240 13147 raft_consensus.cc:2802] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:27.281301 13147 raft_consensus.cc:491] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:27.281330 13147 raft_consensus.cc:3058] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.282302 13147 raft_consensus.cc:513] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.282686 13147 leader_election.cc:290] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Requested vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.283123 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba63065fcf6407a83a8a9ac898e0daf" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2"
I20250905 07:54:27.283249 12767 raft_consensus.cc:3058] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.283205 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba63065fcf6407a83a8a9ac898e0daf" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
I20250905 07:54:27.283342 12910 raft_consensus.cc:3058] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.284199 12767 raft_consensus.cc:2466] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.284267 12910 raft_consensus.cc:2466] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.284545 12987 leader_election.cc:304] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad7908d7d14f30ad901ea6b91babec, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.284762 13147 raft_consensus.cc:2802] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:27.284847 13147 raft_consensus.cc:695] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Becoming Leader. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Running, Role: LEADER
I20250905 07:54:27.284982 13147 consensus_queue.cc:237] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.286628 12606 catalog_manager.cc:5582] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3). New cstate: current_term: 1 leader_uuid: "4cad7908d7d14f30ad901ea6b91babec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } health_report { overall_health: HEALTHY } } }
Sep 05 07:54:27 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.62@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:27.325450 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.317958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:42367 (local address 127.0.0.1:44542)
0905 07:54:27.318237 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.318250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.318365 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.318613 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.318615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.319079 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:27.319289 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.319297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.320004 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.320007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.320418 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.320427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.320539 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.321148 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:27.321162 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:27.322603 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:27.324022 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:27.324032 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:27.324039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:27.324299 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:27.324513 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:27.324516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:27.324518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:27.324560 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:27.324884 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:27.324892 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:27.324995 (+ 103us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.325090 (+ 95us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.325251 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
I20250905 07:54:27.325453 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.318010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.4.49.62:42367)
0905 07:54:27.318138 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.318142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.318317 (+ 175us) server_negotiation.cc:408] Connection header received
0905 07:54:27.318442 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.318465 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.318508 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.318573 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:27.319412 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.319902 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.320521 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.320677 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.322723 (+ 2046us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:27.322746 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:27.322748 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:27.322772 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:27.323918 (+ 1146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:27.324386 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:27.324390 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:27.324392 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:27.324430 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:27.324629 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:27.324632 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:27.324634 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:27.324787 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:27.324869 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.325145 (+ 276us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.325322 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":42}
I20250905 07:54:27.329676 12606 catalog_manager.cc:2232] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:44542:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250905 07:54:27.330458 12606 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:27.335557 12747 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee3f29046844b30bc68269dd1d53cc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6b2e699931404abcc594ac6416006e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:54:27.335623 13033 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee3f29046844b30bc68269dd1d53cc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6b2e699931404abcc594ac6416006e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:54:27.335594 12890 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee3f29046844b30bc68269dd1d53cc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6b2e699931404abcc594ac6416006e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250905 07:54:27.335963 12747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee3f29046844b30bc68269dd1d53cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.335964 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee3f29046844b30bc68269dd1d53cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.335964 12890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee3f29046844b30bc68269dd1d53cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.338508 13130 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: Bootstrap starting.
I20250905 07:54:27.338697 13129 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: Bootstrap starting.
I20250905 07:54:27.339545 13130 tablet_bootstrap.cc:654] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.339545 13129 tablet_bootstrap.cc:654] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.340415 13131 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: Bootstrap starting.
I20250905 07:54:27.340914 13129 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: No bootstrap required, opened a new log
I20250905 07:54:27.341010 13129 ts_tablet_manager.cc:1397] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:27.341434 13131 tablet_bootstrap.cc:654] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.341454 13129 raft_consensus.cc:357] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.341593 13129 raft_consensus.cc:383] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.341632 13129 raft_consensus.cc:738] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.341686 13130 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: No bootstrap required, opened a new log
I20250905 07:54:27.341777 13130 ts_tablet_manager.cc:1397] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:27.341777 13129 consensus_queue.cc:260] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.342072 13129 ts_tablet_manager.cc:1428] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.342293 13130 raft_consensus.cc:357] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.342404 13130 raft_consensus.cc:383] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.342437 13130 raft_consensus.cc:738] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.342563 13130 consensus_queue.cc:260] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.342679 13131 tablet_bootstrap.cc:492] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: No bootstrap required, opened a new log
I20250905 07:54:27.342779 13131 ts_tablet_manager.cc:1397] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:27.342844 13130 ts_tablet_manager.cc:1428] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 07:54:27.342862 13136 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.342999 13136 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.343022 13147 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.343111 13147 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.343266 13131 raft_consensus.cc:357] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.343374 13131 raft_consensus.cc:383] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.343405 13131 raft_consensus.cc:738] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.343528 13131 consensus_queue.cc:260] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.343797 13131 ts_tablet_manager.cc:1428] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.343765 13137 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.343866 13137 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.482175 13137 raft_consensus.cc:491] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:27.482342 13137 raft_consensus.cc:513] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.483443 13137 leader_election.cc:290] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645)
I20250905 07:54:27.486511 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.483684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:33047 (local address 127.4.49.2:44641)
0905 07:54:27.483953 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.483965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.484134 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.484487 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.484489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.484505 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.484741 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.484748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.485456 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.485459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.486054 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.486062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.486183 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.486207 (+ 24us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.486257 (+ 50us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.486321 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20250905 07:54:27.486518 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.484028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:34645 (local address 127.4.49.2:35869)
0905 07:54:27.484238 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.484253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.484345 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.484569 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.484572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.484588 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.484796 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.484802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.485510 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.485513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.486076 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.486083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.486200 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.486221 (+ 21us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.486270 (+ 49us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.486326 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":50,"threads_started":1}
I20250905 07:54:27.486917 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.483750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:44641 (local address 127.4.49.1:33047)
0905 07:54:27.483893 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.483898 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.484085 (+ 187us) server_negotiation.cc:408] Connection header received
0905 07:54:27.484278 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.484282 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.484332 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.484372 (+ 40us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.484860 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.485336 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.486175 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.486637 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.486682 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.486740 (+ 58us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.486802 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250905 07:54:27.486943 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.484188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:35869 (local address 127.4.49.3:34645)
0905 07:54:27.484339 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.484344 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.484359 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:27.484433 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.484436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.484479 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.484540 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.484904 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.485377 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.486188 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.486669 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.486716 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.486770 (+ 54us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.486831 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:54:27.487360 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee3f29046844b30bc68269dd1d53cc1" candidate_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2" is_pre_election: true
I20250905 07:54:27.487506 12767 raft_consensus.cc:2466] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f4da8ecf934af0a7073fc5879e4b5c in term 0.
I20250905 07:54:27.487521 13053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee3f29046844b30bc68269dd1d53cc1" candidate_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad7908d7d14f30ad901ea6b91babec" is_pre_election: true
I20250905 07:54:27.487732 13053 raft_consensus.cc:2466] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f4da8ecf934af0a7073fc5879e4b5c in term 0.
I20250905 07:54:27.487833 12844 leader_election.cc:304] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.488122 13137 raft_consensus.cc:2802] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:27.488186 13137 raft_consensus.cc:491] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:27.488214 13137 raft_consensus.cc:3058] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.489116 13137 raft_consensus.cc:513] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.489478 13137 leader_election.cc:290] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [CANDIDATE]: Term 1 election: Requested vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645)
I20250905 07:54:27.489859 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee3f29046844b30bc68269dd1d53cc1" candidate_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2"
I20250905 07:54:27.489912 13053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee3f29046844b30bc68269dd1d53cc1" candidate_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad7908d7d14f30ad901ea6b91babec"
I20250905 07:54:27.489975 12767 raft_consensus.cc:3058] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.490015 13053 raft_consensus.cc:3058] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.490849 12767 raft_consensus.cc:2466] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f4da8ecf934af0a7073fc5879e4b5c in term 1.
I20250905 07:54:27.490914 13053 raft_consensus.cc:2466] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f4da8ecf934af0a7073fc5879e4b5c in term 1.
I20250905 07:54:27.491156 12844 leader_election.cc:304] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.491326 13137 raft_consensus.cc:2802] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:27.491540 13137 raft_consensus.cc:695] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 LEADER]: Becoming Leader. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Running, Role: LEADER
I20250905 07:54:27.491802 13137 consensus_queue.cc:237] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.493026 13170 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: TxnStatusTablet state changed. Reason: New leader 11f4da8ecf934af0a7073fc5879e4b5c. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.493149 13170 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:54:27.493425 13172 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 07:54:27.493561 13172 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250905 07:54:27.494379 12605 catalog_manager.cc:5582] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2). New cstate: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } health_report { overall_health: UNKNOWN } } }
I20250905 07:54:27.540860 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.537167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.4.49.2:44591)
0905 07:54:27.537447 (+ 280us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.537452 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.537467 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:27.537589 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.537592 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.537656 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.537736 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.538211 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.538786 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.539697 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.539851 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.539906 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.540454 (+ 548us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.540555 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.540660 (+ 105us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.540717 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20250905 07:54:27.540903 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.537027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:44591 (local address 127.0.0.1:58706)
0905 07:54:27.537249 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.537263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.537441 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.537761 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.537764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.537773 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:54:27.538066 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.538075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.538899 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.538904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.539556 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.539568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.539692 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.539764 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.540550 (+ 786us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.540570 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.540645 (+ 75us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.540755 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20250905 07:54:27.543917 12767 raft_consensus.cc:1273] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Refusing update from remote peer 11f4da8ecf934af0a7073fc5879e4b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.544165 13053 raft_consensus.cc:1273] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Refusing update from remote peer 11f4da8ecf934af0a7073fc5879e4b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.544445 13170 consensus_queue.cc:1035] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.544674 13137 consensus_queue.cc:1035] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.546337 13136 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: TxnStatusTablet state changed. Reason: New leader 11f4da8ecf934af0a7073fc5879e4b5c. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.546465 13136 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.546860 13147 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: TxnStatusTablet state changed. Reason: New leader 11f4da8ecf934af0a7073fc5879e4b5c. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.546962 13147 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.548437 13137 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.548543 13137 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:54:27.548632 13143 mvcc.cc:204] Tried to move back new op lower bound from 7196913121454485504 to 7196913121249009664. Current Snapshot: MvccSnapshot[applied={T|T < 7196913121454485504}]
I20250905 07:54:27.551242 13136 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.551371 13136 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P ae18241e0a6c4438889bb2171b1985e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.552533 13170 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.552659 13170 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 11f4da8ecf934af0a7073fc5879e4b5c: This TxnStatusTablet replica's current role is: LEADER
I20250905 07:54:27.553812 13147 tablet_replica.cc:440] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } } }
I20250905 07:54:27.553925 13147 tablet_replica.cc:443] T 6ee3f29046844b30bc68269dd1d53cc1 P 4cad7908d7d14f30ad901ea6b91babec: This TxnStatusTablet replica's current role is: FOLLOWER
I20250905 07:54:27.566872 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.562908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:44591 (local address 127.4.49.3:35325)
0905 07:54:27.563184 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.563198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.563306 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.563620 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.563623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.563632 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:54:27.563846 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.563852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.565444 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.565448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.565920 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.565930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.566050 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.566130 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.566560 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.566567 (+ 7us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.566641 (+ 74us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.566728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20250905 07:54:27.566885 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.563015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35325 (local address 127.4.49.2:44591)
0905 07:54:27.563152 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.563157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.563275 (+ 118us) server_negotiation.cc:408] Connection header received
0905 07:54:27.563402 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.563405 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.563470 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.563567 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.563960 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.565284 (+ 1324us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.566050 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.566273 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.566324 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.566427 (+ 103us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.566506 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.566669 (+ 163us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.566740 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:54:27.574541 12910 raft_consensus.cc:1273] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.575001 13135 consensus_queue.cc:1035] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.575711 12767 raft_consensus.cc:1273] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.576152 13135 consensus_queue.cc:1035] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.583778 13144 mvcc.cc:204] Tried to move back new op lower bound from 7196913121580769280 to 7196913120400556032. Current Snapshot: MvccSnapshot[applied={T|T < 7196913121580769280}]
I20250905 07:54:27.598531 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.595599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:52725 (local address 127.4.49.3:34645)
0905 07:54:27.595725 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.595729 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.595742 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:27.595821 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.595824 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.595872 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.595931 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.596370 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.596943 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.597792 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.598001 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.598049 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.598146 (+ 97us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.598222 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.598320 (+ 98us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.598383 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:54:27.598565 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.595187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:34645 (local address 127.4.49.2:52725)
0905 07:54:27.595614 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.595630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.595735 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.595952 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.595955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.595976 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0905 07:54:27.596257 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.596263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.597049 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.597053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.597667 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.597678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.597783 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.597838 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.598229 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.598235 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.598303 (+ 68us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.598383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
I20250905 07:54:27.628468 12606 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.628630 12606 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.630884 12606 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Sending DeleteTablet for 3 replicas of tablet 6ba63065fcf6407a83a8a9ac898e0daf
I20250905 07:54:27.631608 12747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:47612
I20250905 07:54:27.631680 12890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:58694
I20250905 07:54:27.632033 13187 tablet_replica.cc:331] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: stopping tablet replica
I20250905 07:54:27.632305 13189 tablet_replica.cc:331] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: stopping tablet replica
I20250905 07:54:27.632313 13033 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba63065fcf6407a83a8a9ac898e0daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:35922
I20250905 07:54:27.632524 13189 raft_consensus.cc:2241] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.632728 13190 tablet_replica.cc:331] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: stopping tablet replica
I20250905 07:54:27.632835 13190 raft_consensus.cc:2241] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:27.632707 13187 raft_consensus.cc:2241] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.633038 13187 raft_consensus.cc:2270] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.633037 13190 raft_consensus.cc:2270] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.633167 13189 raft_consensus.cc:2270] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.633957 13190 ts_tablet_manager.cc:1905] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.633981 13189 ts_tablet_manager.cc:1905] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.636016 13187 ts_tablet_manager.cc:1905] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.637125 13189 ts_tablet_manager.cc:1918] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:54:27.637125 13190 ts_tablet_manager.cc:1918] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:54:27.637209 13190 log.cc:1199] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/6ba63065fcf6407a83a8a9ac898e0daf
I20250905 07:54:27.637347 13189 log.cc:1199] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/6ba63065fcf6407a83a8a9ac898e0daf
I20250905 07:54:27.637598 13190 ts_tablet_manager.cc:1939] T 6ba63065fcf6407a83a8a9ac898e0daf P 4cad7908d7d14f30ad901ea6b91babec: Deleting consensus metadata
I20250905 07:54:27.637756 13189 ts_tablet_manager.cc:1939] T 6ba63065fcf6407a83a8a9ac898e0daf P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting consensus metadata
I20250905 07:54:27.638566 12594 catalog_manager.cc:4928] TS 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645): tablet 6ba63065fcf6407a83a8a9ac898e0daf (table test-table [id=9abb8ff828da4517940e29bb1981a955]) successfully deleted
I20250905 07:54:27.638787 12594 catalog_manager.cc:4928] TS 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591): tablet 6ba63065fcf6407a83a8a9ac898e0daf (table test-table [id=9abb8ff828da4517940e29bb1981a955]) successfully deleted
I20250905 07:54:27.639088 13187 ts_tablet_manager.cc:1918] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:54:27.639174 13187 log.cc:1199] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/6ba63065fcf6407a83a8a9ac898e0daf
I20250905 07:54:27.639415 13187 ts_tablet_manager.cc:1939] T 6ba63065fcf6407a83a8a9ac898e0daf P ae18241e0a6c4438889bb2171b1985e2: Deleting consensus metadata
I20250905 07:54:27.640260 12594 catalog_manager.cc:4928] TS ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047): tablet 6ba63065fcf6407a83a8a9ac898e0daf (table test-table [id=9abb8ff828da4517940e29bb1981a955]) successfully deleted
Sep 05 07:54:27 dist-test-slave-mnfb krb5kdc[12559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 05 07:54:27 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:27 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
W20250905 07:54:27.663273 12657 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:27.656667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.4.49.62:42367)
0905 07:54:27.656825 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.656830 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.656965 (+ 135us) server_negotiation.cc:408] Connection header received
0905 07:54:27.657110 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.657113 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.657163 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.657235 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:27.658369 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.658877 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.659759 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.659889 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.663131 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44558: BlockingRecv error: recv got EOF from 127.0.0.1:44558 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 05 07:54:27 dist-test-slave-mnfb krb5kdc[12559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM
I20250905 07:54:27.674077 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.665014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.4.49.62:42367)
0905 07:54:27.665135 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.665139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.665358 (+ 219us) server_negotiation.cc:408] Connection header received
0905 07:54:27.665500 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.665504 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.665557 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.665624 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:27.666418 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.666940 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.667584 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.667782 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.670416 (+ 2634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:27.670447 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:27.670451 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:27.670494 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:27.672258 (+ 1764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:27.672805 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:27.672810 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:27.672812 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:27.672881 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:27.673170 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:27.673174 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:27.673175 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:27.673392 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:27.673529 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.673765 (+ 236us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.673902 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250905 07:54:27.677296 12606 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:27.677840 12606 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:27.683372 12890 tablet_service.cc:1468] Processing CreateTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 (DEFAULT_TABLE table=test-table [id=925d4e5ed94e4085a77c02a2163c3259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.683733 12890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ac93fd38934c96b7e067fbdd07d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.683725 12747 tablet_service.cc:1468] Processing CreateTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 (DEFAULT_TABLE table=test-table [id=925d4e5ed94e4085a77c02a2163c3259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.683971 12747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ac93fd38934c96b7e067fbdd07d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.686323 13033 tablet_service.cc:1468] Processing CreateTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 (DEFAULT_TABLE table=test-table [id=925d4e5ed94e4085a77c02a2163c3259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.686620 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ac93fd38934c96b7e067fbdd07d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.688269 13131 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Bootstrap starting.
I20250905 07:54:27.688417 13129 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Bootstrap starting.
I20250905 07:54:27.689213 13129 tablet_bootstrap.cc:654] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.689219 13131 tablet_bootstrap.cc:654] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.689787 13130 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Bootstrap starting.
I20250905 07:54:27.690380 13131 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: No bootstrap required, opened a new log
I20250905 07:54:27.690479 13131 ts_tablet_manager.cc:1397] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:27.690742 13130 tablet_bootstrap.cc:654] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.690915 13131 raft_consensus.cc:357] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.691020 13131 raft_consensus.cc:383] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.691056 13131 raft_consensus.cc:738] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.691181 13131 consensus_queue.cc:260] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.692284 13130 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: No bootstrap required, opened a new log
I20250905 07:54:27.692381 13130 ts_tablet_manager.cc:1397] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:27.692581 13129 tablet_bootstrap.cc:492] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: No bootstrap required, opened a new log
I20250905 07:54:27.692665 13129 ts_tablet_manager.cc:1397] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250905 07:54:27.692862 13130 raft_consensus.cc:357] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.692967 13130 raft_consensus.cc:383] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.692998 13130 raft_consensus.cc:738] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.691457 13131 ts_tablet_manager.cc:1428] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.693130 13130 consensus_queue.cc:260] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.693120 13129 raft_consensus.cc:357] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.693219 13129 raft_consensus.cc:383] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.693253 13129 raft_consensus.cc:738] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.693403 13130 ts_tablet_manager.cc:1428] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.693364 13129 consensus_queue.cc:260] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.693616 13129 ts_tablet_manager.cc:1428] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.753921 13136 raft_consensus.cc:491] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:27.754070 13136 raft_consensus.cc:513] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.755190 13136 leader_election.cc:290] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591), 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645)
I20250905 07:54:27.758695 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.755846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:34645 (local address 127.4.49.1:38357)
0905 07:54:27.756203 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.756217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.756307 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.756535 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.756537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.756550 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.756876 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.756884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.757687 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.757690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.758306 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.758315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.758429 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.758448 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.758496 (+ 48us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.758552 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":105,"threads_started":1}
I20250905 07:54:27.758693 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.755446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:44591 (local address 127.4.49.1:38771)
0905 07:54:27.755725 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:27.755741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:27.755855 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:27.756164 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:27.756168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:27.756190 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:27.756495 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.756506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.757363 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.757367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:27.758107 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:27.758116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.758215 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.758242 (+ 27us) client_negotiation.cc:769] Sending connection context
0905 07:54:27.758278 (+ 36us) client_negotiation.cc:240] Negotiation successful
0905 07:54:27.758329 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20250905 07:54:27.759028 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.755567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:38771 (local address 127.4.49.2:44591)
0905 07:54:27.755709 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.755712 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.755902 (+ 190us) server_negotiation.cc:408] Connection header received
0905 07:54:27.755967 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.755971 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.756029 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.756141 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.756621 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.757242 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.758241 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.758741 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.758785 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.758848 (+ 63us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.758911 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250905 07:54:27.759192 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.755960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:38357 (local address 127.4.49.3:34645)
0905 07:54:27.756120 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.756125 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.756296 (+ 171us) server_negotiation.cc:408] Connection header received
0905 07:54:27.756385 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.756389 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.756435 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.756524 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:27.756985 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.757568 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.758450 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.758924 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.758965 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.759015 (+ 50us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.759074 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250905 07:54:27.759514 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75ac93fd38934c96b7e067fbdd07d971" candidate_uuid: "ae18241e0a6c4438889bb2171b1985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" is_pre_election: true
I20250905 07:54:27.759603 13053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75ac93fd38934c96b7e067fbdd07d971" candidate_uuid: "ae18241e0a6c4438889bb2171b1985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad7908d7d14f30ad901ea6b91babec" is_pre_election: true
I20250905 07:54:27.759667 12910 raft_consensus.cc:2466] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae18241e0a6c4438889bb2171b1985e2 in term 0.
I20250905 07:54:27.759754 13053 raft_consensus.cc:2466] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae18241e0a6c4438889bb2171b1985e2 in term 0.
I20250905 07:54:27.760042 12701 leader_election.cc:304] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.760316 13136 raft_consensus.cc:2802] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:27.760381 13136 raft_consensus.cc:491] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:27.760425 13136 raft_consensus.cc:3058] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.761353 13136 raft_consensus.cc:513] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.761732 13136 leader_election.cc:290] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591), 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645)
I20250905 07:54:27.762140 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75ac93fd38934c96b7e067fbdd07d971" candidate_uuid: "ae18241e0a6c4438889bb2171b1985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
I20250905 07:54:27.762182 13053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75ac93fd38934c96b7e067fbdd07d971" candidate_uuid: "ae18241e0a6c4438889bb2171b1985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad7908d7d14f30ad901ea6b91babec"
I20250905 07:54:27.762253 12910 raft_consensus.cc:3058] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.762301 13053 raft_consensus.cc:3058] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.763175 12910 raft_consensus.cc:2466] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae18241e0a6c4438889bb2171b1985e2 in term 1.
I20250905 07:54:27.763226 13053 raft_consensus.cc:2466] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae18241e0a6c4438889bb2171b1985e2 in term 1.
I20250905 07:54:27.763500 12701 leader_election.cc:304] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.763695 13136 raft_consensus.cc:2802] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:27.763895 13136 raft_consensus.cc:695] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 1 LEADER]: Becoming Leader. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Running, Role: LEADER
I20250905 07:54:27.764186 13136 consensus_queue.cc:237] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.766767 12607 catalog_manager.cc:5582] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "ae18241e0a6c4438889bb2171b1985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } health_report { overall_health: UNKNOWN } } }
I20250905 07:54:27.793771 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.790640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.4.49.1:33047)
0905 07:54:27.790769 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.790772 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.790785 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:27.790874 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.790876 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.790934 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.791022 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.791369 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.791915 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.792667 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.792819 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.792881 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.793374 (+ 493us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.793494 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.793597 (+ 103us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.793649 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:27.796608 12910 raft_consensus.cc:1273] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Refusing update from remote peer ae18241e0a6c4438889bb2171b1985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.796669 13053 raft_consensus.cc:1273] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Refusing update from remote peer ae18241e0a6c4438889bb2171b1985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:27.797142 13208 consensus_queue.cc:1035] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.797327 13136 consensus_queue.cc:1035] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:27.810186 12607 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.810381 12607 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:27.813248 12607 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Sending DeleteTablet for 3 replicas of tablet 75ac93fd38934c96b7e067fbdd07d971
I20250905 07:54:27.814054 13033 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:35922
I20250905 07:54:27.814082 12747 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:47612
I20250905 07:54:27.814244 13190 tablet_replica.cc:331] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: stopping tablet replica
I20250905 07:54:27.814277 13187 tablet_replica.cc:331] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: stopping tablet replica
I20250905 07:54:27.814368 13190 raft_consensus.cc:2241] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.814419 13187 raft_consensus.cc:2241] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:27.814509 13190 raft_consensus.cc:2270] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.814744 13187 raft_consensus.cc:2270] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.815228 12890 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ac93fd38934c96b7e067fbdd07d971 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:27 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:58694
I20250905 07:54:27.815497 13189 tablet_replica.cc:331] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: stopping tablet replica
I20250905 07:54:27.815620 13189 raft_consensus.cc:2241] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:27.815611 13190 ts_tablet_manager.cc:1905] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.815745 13189 raft_consensus.cc:2270] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:27.815876 13187 ts_tablet_manager.cc:1905] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.815308 12607 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:27.816712 12607 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:27.818284 13190 ts_tablet_manager.cc:1918] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.818367 13190 log.cc:1199] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/75ac93fd38934c96b7e067fbdd07d971
I20250905 07:54:27.818600 13190 ts_tablet_manager.cc:1939] T 75ac93fd38934c96b7e067fbdd07d971 P 4cad7908d7d14f30ad901ea6b91babec: Deleting consensus metadata
I20250905 07:54:27.818565 13187 ts_tablet_manager.cc:1918] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.819191 13187 log.cc:1199] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-0/wal/wals/75ac93fd38934c96b7e067fbdd07d971
I20250905 07:54:27.819432 12594 catalog_manager.cc:4928] TS 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645): tablet 75ac93fd38934c96b7e067fbdd07d971 (table test-table [id=925d4e5ed94e4085a77c02a2163c3259]) successfully deleted
I20250905 07:54:27.819475 13187 ts_tablet_manager.cc:1939] T 75ac93fd38934c96b7e067fbdd07d971 P ae18241e0a6c4438889bb2171b1985e2: Deleting consensus metadata
I20250905 07:54:27.820277 12594 catalog_manager.cc:4928] TS ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047): tablet 75ac93fd38934c96b7e067fbdd07d971 (table test-table [id=925d4e5ed94e4085a77c02a2163c3259]) successfully deleted
I20250905 07:54:27.820703 13189 ts_tablet_manager.cc:1905] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:27.823554 13189 ts_tablet_manager.cc:1918] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:27.823540 12747 tablet_service.cc:1468] Processing CreateTablet for tablet b66c2d07704843209caf649589ec4515 (DEFAULT_TABLE table=test-table [id=6b5b3aec8ad54f1b892cff6a40070d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.823540 13033 tablet_service.cc:1468] Processing CreateTablet for tablet b66c2d07704843209caf649589ec4515 (DEFAULT_TABLE table=test-table [id=6b5b3aec8ad54f1b892cff6a40070d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.823648 13189 log.cc:1199] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/75ac93fd38934c96b7e067fbdd07d971
I20250905 07:54:27.823830 12747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66c2d07704843209caf649589ec4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.823832 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66c2d07704843209caf649589ec4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.823910 13189 ts_tablet_manager.cc:1939] T 75ac93fd38934c96b7e067fbdd07d971 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting consensus metadata
I20250905 07:54:27.826186 13129 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Bootstrap starting.
I20250905 07:54:27.827117 13129 tablet_bootstrap.cc:654] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.827853 12594 catalog_manager.cc:4928] TS 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591): tablet 75ac93fd38934c96b7e067fbdd07d971 (table test-table [id=925d4e5ed94e4085a77c02a2163c3259]) successfully deleted
I20250905 07:54:27.828264 12890 tablet_service.cc:1468] Processing CreateTablet for tablet b66c2d07704843209caf649589ec4515 (DEFAULT_TABLE table=test-table [id=6b5b3aec8ad54f1b892cff6a40070d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:27.828538 12890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66c2d07704843209caf649589ec4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:27.830603 13131 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Bootstrap starting.
I20250905 07:54:27.830729 13130 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: Bootstrap starting.
I20250905 07:54:27.831584 13131 tablet_bootstrap.cc:654] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.831677 13130 tablet_bootstrap.cc:654] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:27.832929 13130 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: No bootstrap required, opened a new log
I20250905 07:54:27.832929 13129 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: No bootstrap required, opened a new log
I20250905 07:54:27.833027 13130 ts_tablet_manager.cc:1397] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:27.833027 13129 ts_tablet_manager.cc:1397] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20250905 07:54:27.833508 13129 raft_consensus.cc:357] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.833498 13130 raft_consensus.cc:357] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.833621 13129 raft_consensus.cc:383] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.833621 13130 raft_consensus.cc:383] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.833653 13129 raft_consensus.cc:738] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.833673 13130 raft_consensus.cc:738] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18241e0a6c4438889bb2171b1985e2, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.833809 13129 consensus_queue.cc:260] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.833817 13130 consensus_queue.cc:260] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.834089 13129 ts_tablet_manager.cc:1428] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.834112 13130 ts_tablet_manager.cc:1428] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 07:54:27.835158 13131 tablet_bootstrap.cc:492] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: No bootstrap required, opened a new log
I20250905 07:54:27.835263 13131 ts_tablet_manager.cc:1397] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250905 07:54:27.835644 13131 raft_consensus.cc:357] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.835723 13131 raft_consensus.cc:383] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:27.835744 13131 raft_consensus.cc:738] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4da8ecf934af0a7073fc5879e4b5c, State: Initialized, Role: FOLLOWER
I20250905 07:54:27.835855 13131 consensus_queue.cc:260] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.836133 13131 ts_tablet_manager.cc:1428] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250905 07:54:27.955765 13147 raft_consensus.cc:491] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:27.955938 13147 raft_consensus.cc:513] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.956480 13147 leader_election.cc:290] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.956977 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b66c2d07704843209caf649589ec4515" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2" is_pre_election: true
I20250905 07:54:27.957046 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b66c2d07704843209caf649589ec4515" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" is_pre_election: true
I20250905 07:54:27.957126 12767 raft_consensus.cc:2466] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.957203 12910 raft_consensus.cc:2466] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 0.
I20250905 07:54:27.957494 12987 leader_election.cc:304] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad7908d7d14f30ad901ea6b91babec, ae18241e0a6c4438889bb2171b1985e2; no voters:
I20250905 07:54:27.957687 13147 raft_consensus.cc:2802] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:27.957749 13147 raft_consensus.cc:491] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:27.957791 13147 raft_consensus.cc:3058] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.958729 13147 raft_consensus.cc:513] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.959066 13147 leader_election.cc:290] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Requested vote from peers ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047), 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591)
I20250905 07:54:27.959517 12910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b66c2d07704843209caf649589ec4515" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4da8ecf934af0a7073fc5879e4b5c"
I20250905 07:54:27.959519 12767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b66c2d07704843209caf649589ec4515" candidate_uuid: "4cad7908d7d14f30ad901ea6b91babec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18241e0a6c4438889bb2171b1985e2"
I20250905 07:54:27.959640 12910 raft_consensus.cc:3058] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.959646 12767 raft_consensus.cc:3058] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:27.960608 12910 raft_consensus.cc:2466] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.960901 12767 raft_consensus.cc:2466] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad7908d7d14f30ad901ea6b91babec in term 1.
I20250905 07:54:27.960920 12987 leader_election.cc:304] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4da8ecf934af0a7073fc5879e4b5c, 4cad7908d7d14f30ad901ea6b91babec; no voters:
I20250905 07:54:27.961095 13147 raft_consensus.cc:2802] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:27.961182 13147 raft_consensus.cc:695] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Becoming Leader. State: Replica: 4cad7908d7d14f30ad901ea6b91babec, State: Running, Role: LEADER
I20250905 07:54:27.961328 13147 consensus_queue.cc:237] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } }
I20250905 07:54:27.963188 12607 catalog_manager.cc:5582] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3). New cstate: current_term: 1 leader_uuid: "4cad7908d7d14f30ad901ea6b91babec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cad7908d7d14f30ad901ea6b91babec" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 34645 } health_report { overall_health: HEALTHY } } }
I20250905 07:54:27.996415 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:27.993590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.4.49.3:34645)
0905 07:54:27.993726 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:27.993731 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:27.993745 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:27.993847 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:27.993850 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:27.993904 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:27.993982 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:27.994281 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.994922 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:27.995657 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:27.995858 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:27.995912 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:27.995999 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:27.996120 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:27.996204 (+ 84us) server_negotiation.cc:300] Negotiation successful
0905 07:54:27.996270 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:54:28.003149 12910 raft_consensus.cc:1273] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:28.003149 12767 raft_consensus.cc:1273] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Refusing update from remote peer 4cad7908d7d14f30ad901ea6b91babec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:28.003538 13139 consensus_queue.cc:1035] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f4da8ecf934af0a7073fc5879e4b5c" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:28.003718 13147 consensus_queue.cc:1035] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae18241e0a6c4438889bb2171b1985e2" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 33047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:28.006431 13142 mvcc.cc:204] Tried to move back new op lower bound from 7196913123337228288 to 7196913123171147776. Current Snapshot: MvccSnapshot[applied={T|T < 7196913123337228288}]
I20250905 07:54:28.038024 12607 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:28.038189 12607 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:28.040472 12607 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ed0b16e84e324b309baa85cd3d457dd6: Sending DeleteTablet for 3 replicas of tablet b66c2d07704843209caf649589ec4515
I20250905 07:54:28.041253 12747 tablet_service.cc:1515] Processing DeleteTablet for tablet b66c2d07704843209caf649589ec4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:28 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:47612
I20250905 07:54:28.041478 13187 tablet_replica.cc:331] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2: stopping tablet replica
I20250905 07:54:28.041603 13187 raft_consensus.cc:2241] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:28.041751 13187 raft_consensus.cc:2270] T b66c2d07704843209caf649589ec4515 P ae18241e0a6c4438889bb2171b1985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:28.041890 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12682
I20250905 07:54:28.042048 13033 tablet_service.cc:1515] Processing DeleteTablet for tablet b66c2d07704843209caf649589ec4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:28 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:35922
I20250905 07:54:28.042668 13190 tablet_replica.cc:331] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: stopping tablet replica
I20250905 07:54:28.042786 13190 raft_consensus.cc:2241] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:28.043008 13190 raft_consensus.cc:2270] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:28.043735 12890 tablet_service.cc:1515] Processing DeleteTablet for tablet b66c2d07704843209caf649589ec4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:28 UTC) from {username='oryx', principal='oryx/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:58694
I20250905 07:54:28.043900 13190 ts_tablet_manager.cc:1905] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:28.043944 13189 tablet_replica.cc:331] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: stopping tablet replica
I20250905 07:54:28.044081 13189 raft_consensus.cc:2241] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:28.044222 13189 raft_consensus.cc:2270] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:28.045111 13189 ts_tablet_manager.cc:1905] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:28.046537 13190 ts_tablet_manager.cc:1918] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:54:28.046635 13190 log.cc:1199] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/b66c2d07704843209caf649589ec4515
I20250905 07:54:28.046880 13190 ts_tablet_manager.cc:1939] T b66c2d07704843209caf649589ec4515 P 4cad7908d7d14f30ad901ea6b91babec: Deleting consensus metadata
I20250905 07:54:28.047675 13189 ts_tablet_manager.cc:1918] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250905 07:54:28.047763 13189 log.cc:1199] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/b66c2d07704843209caf649589ec4515
I20250905 07:54:28.047772 12594 catalog_manager.cc:4928] TS 4cad7908d7d14f30ad901ea6b91babec (127.4.49.3:34645): tablet b66c2d07704843209caf649589ec4515 (table test-table [id=6b5b3aec8ad54f1b892cff6a40070d00]) successfully deleted
I20250905 07:54:28.048014 13189 ts_tablet_manager.cc:1939] T b66c2d07704843209caf649589ec4515 P 11f4da8ecf934af0a7073fc5879e4b5c: Deleting consensus metadata
I20250905 07:54:28.048768 12594 catalog_manager.cc:4928] TS 11f4da8ecf934af0a7073fc5879e4b5c (127.4.49.2:44591): tablet b66c2d07704843209caf649589ec4515 (table test-table [id=6b5b3aec8ad54f1b892cff6a40070d00]) successfully deleted
W20250905 07:54:28.049943 12594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:33047 (error 108)
I20250905 07:54:28.050230 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12825
I20250905 07:54:28.051630 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:28.051219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:33047 (local address 127.0.0.1:49572)
0905 07:54:28.051458 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:28.051529 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:33047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
W20250905 07:54:28.051865 12594 catalog_manager.cc:4671] TS ae18241e0a6c4438889bb2171b1985e2 (127.4.49.1:33047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b66c2d07704843209caf649589ec4515: Network error: Client connection negotiation failed: client connection to 127.4.49.1:33047: connect: Connection refused (error 111)
I20250905 07:54:28.057195 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12968
I20250905 07:54:28.063572 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 12575
2025-09-05T07:54:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4090 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:28 dist-test-slave-mnfb krb5kdc[13224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:28 dist-test-slave-mnfb krb5kdc[13224](info): set up 2 sockets
Sep 05 07:54:28 dist-test-slave-mnfb krb5kdc[13224](info): commencing operation
krb5kdc: starting...
W20250905 07:54:29.903262 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.817s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:29 dist-test-slave-mnfb krb5kdc[13224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:30.040094 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:33915
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:38449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:30.151831 13240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:30.152144 13240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:30.152227 13240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:30.155714 13240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:30.155794 13240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:30.155819 13240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:30.155851 13240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:30.155882 13240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:30.160761 13240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:38449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:30.161918 13240 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:30.162846 13240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:30.168790 13245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:30.168907 13240 server_base.cc:1047] running on GCE node
W20250905 07:54:30.168795 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:30.168790 13246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:30.169637 13240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:30.170629 13240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:30.171793 13240 hybrid_clock.cc:648] HybridClock initialized: now 1757058870171776 us; error 34 us; skew 500 ppm
W20250905 07:54:30.176545 13240 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-09-05T07:54:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2113 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:30 dist-test-slave-mnfb krb5kdc[13253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:30 dist-test-slave-mnfb krb5kdc[13253](info): set up 2 sockets
Sep 05 07:54:30 dist-test-slave-mnfb krb5kdc[13253](info): commencing operation
krb5kdc: starting...
W20250905 07:54:32.015156 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:32.150362 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38767
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:38767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:32.265341 13270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:32.265607 13270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:32.265687 13270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:32.269193 13270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:32.269286 13270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:32.269317 13270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:32.269343 13270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:32.269366 13270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:32.274111 13270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:38767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:32.275314 13270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:32.276285 13270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:32.282444 13278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:32.282444 13275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.282538 13270 server_base.cc:1047] running on GCE node
W20250905 07:54:32.282454 13276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.283166 13270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:32.284219 13270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:32.285432 13270 hybrid_clock.cc:648] HybridClock initialized: now 1757058872285409 us; error 39 us; skew 500 ppm
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:32.288429 13270 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:54:32.289678 13270 webserver.cc:480] Webserver started at http://127.4.49.62:36917/ using document root <none> and password file <none>
I20250905 07:54:32.290262 13270 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:32.290313 13270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:32.290493 13270 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:32.292462 13270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "779e734efcec4bfeb96438d7452599ad"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "de2d7d91ea8fdc29d32bc05967d34512"
server_key_iv: "d2e14bf1996c9c41e5fe288b70cf7bc5"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.292989 13270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "779e734efcec4bfeb96438d7452599ad"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "de2d7d91ea8fdc29d32bc05967d34512"
server_key_iv: "d2e14bf1996c9c41e5fe288b70cf7bc5"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.296813 13270 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250905 07:54:32.299248 13285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.300443 13270 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:32.300571 13270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "779e734efcec4bfeb96438d7452599ad"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "de2d7d91ea8fdc29d32bc05967d34512"
server_key_iv: "d2e14bf1996c9c41e5fe288b70cf7bc5"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.300681 13270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:32.315546 13270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:32.322600 13270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:32.322799 13270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:32.330250 13337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:38767 every 8 connection(s)
I20250905 07:54:32.330251 13270 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:38767
I20250905 07:54:32.331432 13270 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:32.334662 13338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:32.336793 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13270
I20250905 07:54:32.336933 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:32.337189 4292 external_mini_cluster.cc:1468] Setting key f40757bbc0a5f603f901ea734df96f38
I20250905 07:54:32.341646 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Bootstrap starting.
I20250905 07:54:32.344405 13338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Neither blocks nor log segments found. Creating new log.
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:32.345235 13338 log.cc:826] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:32.347561 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: No bootstrap required, opened a new log
I20250905 07:54:32.350905 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:32.338664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.4.49.62:38767)
0905 07:54:32.339083 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:32.339093 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:32.339130 (+ 37us) server_negotiation.cc:408] Connection header received
0905 07:54:32.339671 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:32.339689 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:32.340010 (+ 321us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:32.340360 (+ 350us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:32.341373 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.342198 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:32.343060 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.343266 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:32.345891 (+ 2625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:32.345939 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:32.345954 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:32.345991 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:32.348123 (+ 2132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.348582 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.348588 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.348594 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.348707 (+ 113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.348931 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.348934 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.348936 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.349383 (+ 447us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:32.349584 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:32.349827 (+ 243us) server_negotiation.cc:300] Negotiation successful
0905 07:54:32.350116 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20250905 07:54:32.350821 13338 raft_consensus.cc:357] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:32.351045 13338 raft_consensus.cc:383] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:32.351118 13338 raft_consensus.cc:738] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779e734efcec4bfeb96438d7452599ad, State: Initialized, Role: FOLLOWER
I20250905 07:54:32.351613 13338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:32.351775 13338 raft_consensus.cc:397] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:32.351838 13338 raft_consensus.cc:491] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:32.351958 13338 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:32.352902 13338 raft_consensus.cc:513] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:32.353294 13338 leader_election.cc:304] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779e734efcec4bfeb96438d7452599ad; no voters:
I20250905 07:54:32.353600 13338 leader_election.cc:290] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:32.353715 13343 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:32.353906 13343 raft_consensus.cc:695] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 LEADER]: Becoming Leader. State: Replica: 779e734efcec4bfeb96438d7452599ad, State: Running, Role: LEADER
I20250905 07:54:32.354221 13343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:32.354748 13338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:32.355973 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779e734efcec4bfeb96438d7452599ad. Latest consensus state: current_term: 1 leader_uuid: "779e734efcec4bfeb96438d7452599ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } } }
I20250905 07:54:32.356145 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:32.356417 13344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "779e734efcec4bfeb96438d7452599ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } } }
I20250905 07:54:32.356510 13344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:32.357177 13352 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:32.359848 13352 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:32.365725 13352 catalog_manager.cc:1349] Generated new cluster ID: 48fb25e0a24946f49e832eb47fed41d3
I20250905 07:54:32.365830 13352 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:32.384334 13352 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:32.385318 13352 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:32.394366 13352 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Generated new TSK 0
I20250905 07:54:32.395134 13352 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:32.447649 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:38767
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:32.559532 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:32.559792 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:32.559854 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:32.563418 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:32.563506 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:32.563601 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:32.568264 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:32.569411 13366 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:32.570309 13366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:32.577418 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.577546 13366 server_base.cc:1047] running on GCE node
W20250905 07:54:32.577368 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:32.577618 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.578032 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:32.578632 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:32.579787 13366 hybrid_clock.cc:648] HybridClock initialized: now 1757058872579769 us; error 36 us; skew 500 ppm
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:32.582594 13366 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:54:32.583789 13366 webserver.cc:480] Webserver started at http://127.4.49.1:34257/ using document root <none> and password file <none>
I20250905 07:54:32.584420 13366 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:32.584484 13366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:32.584679 13366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:32.586503 13366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "f1f915e09d244dc1a0c89b5cd674006a"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "a4dcabcf1cd39fb951e9fd0fc3415639"
server_key_iv: "67fa31634fff0637ee01c70de732ad75"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.587006 13366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f915e09d244dc1a0c89b5cd674006a"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "a4dcabcf1cd39fb951e9fd0fc3415639"
server_key_iv: "67fa31634fff0637ee01c70de732ad75"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.590619 13366 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250905 07:54:32.593077 13381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.594173 13366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:32.594308 13366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "f1f915e09d244dc1a0c89b5cd674006a"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "a4dcabcf1cd39fb951e9fd0fc3415639"
server_key_iv: "67fa31634fff0637ee01c70de732ad75"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.594410 13366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:32.606020 13366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:32.609146 13366 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:32.609339 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:32.609922 13366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:32.610837 13366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:32.610888 13366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.610942 13366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:32.610968 13366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.620589 13366 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:41271
I20250905 07:54:32.620662 13495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:41271 every 8 connection(s)
I20250905 07:54:32.621624 13366 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:32.623689 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13366
I20250905 07:54:32.623814 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:32.624081 4292 external_mini_cluster.cc:1468] Setting key 8ef681e536f9b5937bc3d725e96b7c13
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:54:32.634778 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:32.623535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:57475 (local address 127.4.49.62:38767)
0905 07:54:32.623705 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:32.623710 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:32.624547 (+ 837us) server_negotiation.cc:408] Connection header received
0905 07:54:32.625474 (+ 927us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:32.625477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:32.625533 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:32.625604 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:32.626960 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.627470 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:32.628657 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.628855 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:32.631153 (+ 2298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:32.631176 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:32.631179 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:32.631210 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:32.632701 (+ 1491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.633206 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.633210 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.633212 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.633263 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.633586 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.633590 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.633592 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.633779 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:32.633869 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:32.634443 (+ 574us) server_negotiation.cc:300] Negotiation successful
0905 07:54:32.634588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:54:32.635479 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:32.623840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:57475)
0905 07:54:32.624372 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:32.624423 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:32.625275 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:32.625760 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:32.625769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:32.626169 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:32.626786 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:32.626804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.627587 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:32.627591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:32.628526 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:32.628536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.628765 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:32.629444 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:32.629482 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:32.630983 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:32.632829 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:32.632836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:32.632851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:32.633104 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:32.633346 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:32.633382 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:32.633383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:32.633495 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:32.633884 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:32.633893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:32.634137 (+ 244us) client_negotiation.cc:769] Sending connection context
0905 07:54:32.634389 (+ 252us) client_negotiation.cc:240] Negotiation successful
0905 07:54:32.634756 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20250905 07:54:32.636806 13496 heartbeater.cc:344] Connected to a master server at 127.4.49.62:38767
I20250905 07:54:32.637063 13496 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:32.637630 13496 heartbeater.cc:507] Master 127.4.49.62:38767 requested a full tablet report, sending...
I20250905 07:54:32.639241 13302 ts_manager.cc:194] Registered new tserver with Master: f1f915e09d244dc1a0c89b5cd674006a (127.4.49.1:41271)
I20250905 07:54:32.640980 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:57475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:32.670133 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:38767
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:32.779201 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:32.779455 13503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:32.779515 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:32.782979 13503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:32.783069 13503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:32.783154 13503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:32.787719 13503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:32.788859 13503 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:32.789731 13503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:32.796613 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.796664 13503 server_base.cc:1047] running on GCE node
W20250905 07:54:32.796689 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:32.796638 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:32.797248 13503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:32.797883 13503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:32.799034 13503 hybrid_clock.cc:648] HybridClock initialized: now 1757058872799018 us; error 32 us; skew 500 ppm
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:32.801928 13503 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:54:32.803117 13503 webserver.cc:480] Webserver started at http://127.4.49.2:46527/ using document root <none> and password file <none>
I20250905 07:54:32.803714 13503 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:32.803767 13503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:32.803962 13503 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:32.805953 13503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "6b2d2051649a45fa8587b32ffad0adbf"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "4b7ca25d27882fd38fba83b2ce18a9c6"
server_key_iv: "72fc7d892738d0e960908e4cbb3e213e"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.806464 13503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "6b2d2051649a45fa8587b32ffad0adbf"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "4b7ca25d27882fd38fba83b2ce18a9c6"
server_key_iv: "72fc7d892738d0e960908e4cbb3e213e"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.810228 13503 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:32.812767 13518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.813933 13503 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:32.814065 13503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "6b2d2051649a45fa8587b32ffad0adbf"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "4b7ca25d27882fd38fba83b2ce18a9c6"
server_key_iv: "72fc7d892738d0e960908e4cbb3e213e"
server_key_version: "encryptionkey@0"
I20250905 07:54:32.814184 13503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:32.827016 13503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:32.830781 13503 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:32.830988 13503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:32.831660 13503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:32.832659 13503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:32.832715 13503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.832770 13503 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:32.832798 13503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:32.841398 13503 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:36645
I20250905 07:54:32.841421 13632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:36645 every 8 connection(s)
I20250905 07:54:32.842415 13503 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:32.846062 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13503
I20250905 07:54:32.846177 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:32.846429 4292 external_mini_cluster.cc:1468] Setting key 615688770da205f9a590a998e43283ec
Sep 05 07:54:32 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058872, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:54:32.855947 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:32.844264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:51489 (local address 127.4.49.62:38767)
0905 07:54:32.844404 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:32.844408 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:32.845226 (+ 818us) server_negotiation.cc:408] Connection header received
0905 07:54:32.846113 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:32.846117 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:32.846179 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:32.846253 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:32.847900 (+ 1647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.848445 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:32.849604 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.849795 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:32.852359 (+ 2564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:32.852383 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:32.852386 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:32.852415 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:32.853884 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.854424 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.854427 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.854429 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.854482 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:32.854775 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:32.854779 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:32.854781 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:32.854974 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:32.855065 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:32.855619 (+ 554us) server_negotiation.cc:300] Negotiation successful
0905 07:54:32.855773 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250905 07:54:32.856792 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:32.844570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:51489)
0905 07:54:32.845059 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:32.845100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:32.845894 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:32.846576 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:32.846589 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:32.847006 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:32.847750 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:32.847768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.848577 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:32.848580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:32.849467 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:32.849477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:32.849703 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:32.850418 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:32.850458 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:32.852174 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:32.854027 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:32.854035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:32.854053 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:32.854317 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:32.854569 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:32.854572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:32.854574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:32.854684 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:32.855076 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:32.855085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:32.855349 (+ 264us) client_negotiation.cc:769] Sending connection context
0905 07:54:32.855580 (+ 231us) client_negotiation.cc:240] Negotiation successful
0905 07:54:32.856007 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:54:32.858081 13633 heartbeater.cc:344] Connected to a master server at 127.4.49.62:38767
I20250905 07:54:32.858348 13633 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:32.858937 13633 heartbeater.cc:507] Master 127.4.49.62:38767 requested a full tablet report, sending...
I20250905 07:54:32.860121 13302 ts_manager.cc:194] Registered new tserver with Master: 6b2d2051649a45fa8587b32ffad0adbf (127.4.49.2:36645)
I20250905 07:54:32.860765 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:51489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:32.893291 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:38767
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:33.000996 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:33.001245 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:33.001327 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:33.004979 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:33.005069 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:33.005173 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:33.009711 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:33.010864 13640 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:33.011750 13640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:33.018503 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.018520 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.018796 13640 server_base.cc:1047] running on GCE node
W20250905 07:54:33.018932 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.019395 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:33.020141 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:33.021330 13640 hybrid_clock.cc:648] HybridClock initialized: now 1757058873021308 us; error 40 us; skew 500 ppm
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:33.024924 13640 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:54:33.026312 13640 webserver.cc:480] Webserver started at http://127.4.49.3:44267/ using document root <none> and password file <none>
I20250905 07:54:33.027045 13640 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:33.027112 13640 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:33.027338 13640 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:33.029808 13640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "d6c901aa80974660b7df2ba1cb2215d7"
format_stamp: "Formatted at 2025-09-05 07:54:33 on dist-test-slave-mnfb"
server_key: "b52c1e3021d91254ce397abae4f3752d"
server_key_iv: "52f5979409391bd1bb9f9d51fdab864b"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.030407 13640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "d6c901aa80974660b7df2ba1cb2215d7"
format_stamp: "Formatted at 2025-09-05 07:54:33 on dist-test-slave-mnfb"
server_key: "b52c1e3021d91254ce397abae4f3752d"
server_key_iv: "52f5979409391bd1bb9f9d51fdab864b"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.035091 13640 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250905 07:54:33.038130 13655 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.039445 13640 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:54:33.039582 13640 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "d6c901aa80974660b7df2ba1cb2215d7"
format_stamp: "Formatted at 2025-09-05 07:54:33 on dist-test-slave-mnfb"
server_key: "b52c1e3021d91254ce397abae4f3752d"
server_key_iv: "52f5979409391bd1bb9f9d51fdab864b"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.039700 13640 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:33.069206 13640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:33.073776 13640 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:33.073968 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:33.074527 13640 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:33.075438 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:33.075500 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.075555 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:33.075584 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.084190 13640 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:35395
I20250905 07:54:33.084221 13769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:35395 every 8 connection(s)
I20250905 07:54:33.085140 13640 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:33.089378 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13640
I20250905 07:54:33.089501 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:33.089725 4292 external_mini_cluster.cc:1468] Setting key 9f06341a0bf3387ee4135090ced95f07
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:33.097363 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.086908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:54501 (local address 127.4.49.62:38767)
0905 07:54:33.087043 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.087047 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.087744 (+ 697us) server_negotiation.cc:408] Connection header received
0905 07:54:33.088648 (+ 904us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.088652 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.088712 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.088776 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.090170 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.090746 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.091861 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.092041 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.094231 (+ 2190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:33.094248 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:33.094251 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:33.094281 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:33.095420 (+ 1139us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:33.095939 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:33.095942 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:33.095944 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:33.095992 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:33.096280 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:33.096284 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:33.096285 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:33.096447 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:33.096535 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:33.097062 (+ 527us) server_negotiation.cc:300] Negotiation successful
0905 07:54:33.097198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:54:33.098006 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.087161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:54501)
0905 07:54:33.087585 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.087623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.088438 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.088929 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.088938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.089370 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.090007 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.090023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.090871 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.090874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.091739 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.091748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.091953 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.092582 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.092619 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.094079 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:33.095557 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:33.095564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:33.095578 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:33.095825 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:33.096094 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:33.096097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:33.096100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:33.096197 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:33.096553 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:33.096560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:33.096809 (+ 249us) client_negotiation.cc:769] Sending connection context
0905 07:54:33.096995 (+ 186us) client_negotiation.cc:240] Negotiation successful
0905 07:54:33.097285 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20250905 07:54:33.099107 13770 heartbeater.cc:344] Connected to a master server at 127.4.49.62:38767
I20250905 07:54:33.099336 13770 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:33.099853 13770 heartbeater.cc:507] Master 127.4.49.62:38767 requested a full tablet report, sending...
I20250905 07:54:33.100844 13302 ts_manager.cc:194] Registered new tserver with Master: d6c901aa80974660b7df2ba1cb2215d7 (127.4.49.3:35395)
I20250905 07:54:33.101379 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:54501
I20250905 07:54:33.103192 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.1.keytab.
Entry for principal oryx/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.1.keytab.
WARNING: no policy specified for oryx/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.2.keytab.
Entry for principal oryx/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.2.keytab.
WARNING: no policy specified for oryx/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.3.keytab.
Entry for principal oryx/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.3.keytab.
I20250905 07:54:33.164269 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13366
I20250905 07:54:33.170173 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13503
I20250905 07:54:33.175354 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13640
I20250905 07:54:33.180452 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13270
I20250905 07:54:33.186060 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38767
--webserver_interface=127.4.49.62
--webserver_port=36917
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:38767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:33.293512 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:33.293771 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:33.293830 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:33.297232 13779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:33.297317 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:33.297343 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:33.297375 13779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:33.297430 13779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:33.301957 13779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:38767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=36917
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:33.303098 13779 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:33.303982 13779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:33.309541 13784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.309542 13785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.309593 13779 server_base.cc:1047] running on GCE node
W20250905 07:54:33.309541 13787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.310267 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:33.311170 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:33.312305 13779 hybrid_clock.cc:648] HybridClock initialized: now 1757058873312286 us; error 36 us; skew 500 ppm
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:33.315083 13779 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:54:33.316257 13779 webserver.cc:480] Webserver started at http://127.4.49.62:36917/ using document root <none> and password file <none>
I20250905 07:54:33.316823 13779 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:33.316890 13779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:33.320519 13779 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250905 07:54:33.322628 13794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.323693 13779 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:33.323823 13779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "779e734efcec4bfeb96438d7452599ad"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "de2d7d91ea8fdc29d32bc05967d34512"
server_key_iv: "d2e14bf1996c9c41e5fe288b70cf7bc5"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.324287 13779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:33.346670 13779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:33.349706 13779 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:33.349886 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:33.357439 13779 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:38767
I20250905 07:54:33.357416 13846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:38767 every 8 connection(s)
I20250905 07:54:33.358628 13779 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:33.361554 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13779
I20250905 07:54:33.362080 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:41271
--local_ip_for_outbound_sockets=127.4.49.1
--tserver_master_addrs=127.4.49.62:38767
--webserver_port=34257
--webserver_interface=127.4.49.1
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.49.1
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250905 07:54:33.363811 13847 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:54:33.365713 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Bootstrap starting.
I20250905 07:54:33.382508 13847 log.cc:826] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:33.389225 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:54:33.389616 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Bootstrap complete.
I20250905 07:54:33.392985 13847 raft_consensus.cc:357] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:33.393309 13847 raft_consensus.cc:738] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779e734efcec4bfeb96438d7452599ad, State: Initialized, Role: FOLLOWER
I20250905 07:54:33.393796 13847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:33.393915 13847 raft_consensus.cc:397] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:33.393973 13847 raft_consensus.cc:491] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:33.394080 13847 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:54:33.395700 13847 raft_consensus.cc:513] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:33.396045 13847 leader_election.cc:304] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779e734efcec4bfeb96438d7452599ad; no voters:
I20250905 07:54:33.396379 13847 leader_election.cc:290] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 07:54:33.396559 13851 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 2 FOLLOWER]: Leader election won for term 2
I20250905 07:54:33.396857 13851 raft_consensus.cc:695] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [term 2 LEADER]: Becoming Leader. State: Replica: 779e734efcec4bfeb96438d7452599ad, State: Running, Role: LEADER
I20250905 07:54:33.397224 13851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } }
I20250905 07:54:33.397523 13847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:33.398589 13852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "779e734efcec4bfeb96438d7452599ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } } }
I20250905 07:54:33.398718 13852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:33.398967 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779e734efcec4bfeb96438d7452599ad. Latest consensus state: current_term: 2 leader_uuid: "779e734efcec4bfeb96438d7452599ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779e734efcec4bfeb96438d7452599ad" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38767 } } }
I20250905 07:54:33.399039 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:33.399502 13860 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:33.402995 13860 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:33.403664 13860 catalog_manager.cc:1261] Loaded cluster ID: 48fb25e0a24946f49e832eb47fed41d3
I20250905 07:54:33.403728 13860 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:33.406176 13860 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:33.406917 13860 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 779e734efcec4bfeb96438d7452599ad: Loaded TSK: 0
I20250905 07:54:33.408264 13860 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 07:54:33.479485 13849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:33.479768 13849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:33.479835 13849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:33.483419 13849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:33.483515 13849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:33.483613 13849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:33.488319 13849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.1.keytab
--principal=oryx/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:41271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=34257
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:33.489446 13849 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:33.490340 13849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:33.497238 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.497509 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.497244 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.497684 13849 server_base.cc:1047] running on GCE node
I20250905 07:54:33.498139 13849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:33.498842 13849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:33.500016 13849 hybrid_clock.cc:648] HybridClock initialized: now 1757058873500013 us; error 57 us; skew 500 ppm
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:33.503568 13849 init.cc:377] Logged in from keytab as oryx/127.4.49.1@KRBTEST.COM (short username oryx)
I20250905 07:54:33.505070 13849 webserver.cc:480] Webserver started at http://127.4.49.1:34257/ using document root <none> and password file <none>
I20250905 07:54:33.505807 13849 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:33.505903 13849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:33.510772 13849 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:33.513383 13883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.514772 13849 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:54:33.514922 13849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "f1f915e09d244dc1a0c89b5cd674006a"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "a4dcabcf1cd39fb951e9fd0fc3415639"
server_key_iv: "67fa31634fff0637ee01c70de732ad75"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.515497 13849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:33.562939 13849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:33.566738 13849 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:33.566951 13849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:33.567909 13849 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:33.569221 13849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:33.569298 13849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.569357 13849 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:33.569396 13849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.580523 13849 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:41271
I20250905 07:54:33.580626 13998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:41271 every 8 connection(s)
I20250905 07:54:33.581887 13849 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:33.589252 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 13849
I20250905 07:54:33.589787 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:36645
--local_ip_for_outbound_sockets=127.4.49.2
--tserver_master_addrs=127.4.49.62:38767
--webserver_port=46527
--webserver_interface=127.4.49.2
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.49.2
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.594331 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.584332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:50311)
0905 07:54:33.584782 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.584820 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.585607 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.586712 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.586721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.587191 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.587761 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.587775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.588890 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.588894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.589450 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.589461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.589693 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.590914 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.590957 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.593672 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
W20250905 07:54:33.594703 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.584137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:50311 (local address 127.4.49.62:38767)
0905 07:54:33.584558 (+ 421us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.584571 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.584932 (+ 361us) server_negotiation.cc:408] Connection header received
0905 07:54:33.586022 (+ 1090us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.586033 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.586315 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.586613 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.587926 (+ 1313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.588754 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.589580 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.589801 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.593972 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:50311: BlockingRecv error: recv got EOF from 127.4.49.1:50311 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
W20250905 07:54:33.594887 13999 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:38767 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.49.62:38767: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.600351 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.595393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:52537)
0905 07:54:33.595520 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.595535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.595617 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.595857 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.595859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.595998 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.596219 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.596226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.597287 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.597291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.597736 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.597745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.597848 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.598347 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.598361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.600245 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20250905 07:54:33.600898 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.595470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:52537 (local address 127.4.49.62:38767)
0905 07:54:33.595611 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.595615 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.595632 (+ 17us) server_negotiation.cc:408] Connection header received
0905 07:54:33.595687 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.595690 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.595757 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.595846 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.596330 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.597108 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.597851 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.598053 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.600413 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:52537: BlockingRecv error: recv got EOF from 127.4.49.1:52537 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.605576 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.600955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:45607)
0905 07:54:33.601126 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.601141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.601223 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.601457 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.601460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.601584 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.601758 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.601764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.602759 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.602762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.603119 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.603126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.603211 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.603684 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.603697 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.605463 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20250905 07:54:33.605835 13999 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250905 07:54:33.605861 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.601031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:45607 (local address 127.4.49.62:38767)
0905 07:54:33.601138 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.601142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.601182 (+ 40us) server_negotiation.cc:408] Connection header received
0905 07:54:33.601303 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.601306 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.601352 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.601431 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.601870 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.602640 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.603217 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.603407 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.605658 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:45607: BlockingRecv error: recv got EOF from 127.4.49.1:45607 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250905 07:54:33.704715 14003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:33.704962 14003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:33.705032 14003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:33.708622 14003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:33.708706 14003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:33.708792 14003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:33.713392 14003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.2.keytab
--principal=oryx/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:36645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=46527
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:33.714557 14003 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:33.715435 14003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:33.722195 14008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.722319 14011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.722195 14009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.722419 14003 server_base.cc:1047] running on GCE node
I20250905 07:54:33.722899 14003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:33.723537 14003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:33.724710 14003 hybrid_clock.cc:648] HybridClock initialized: now 1757058873724693 us; error 32 us; skew 500 ppm
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:33.727447 14003 init.cc:377] Logged in from keytab as oryx/127.4.49.2@KRBTEST.COM (short username oryx)
I20250905 07:54:33.728695 14003 webserver.cc:480] Webserver started at http://127.4.49.2:46527/ using document root <none> and password file <none>
I20250905 07:54:33.729282 14003 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:33.729355 14003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:33.733178 14003 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250905 07:54:33.735317 14018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.736416 14003 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:33.736537 14003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "6b2d2051649a45fa8587b32ffad0adbf"
format_stamp: "Formatted at 2025-09-05 07:54:32 on dist-test-slave-mnfb"
server_key: "4b7ca25d27882fd38fba83b2ce18a9c6"
server_key_iv: "72fc7d892738d0e960908e4cbb3e213e"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.736961 14003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:33.748893 14003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:33.751936 14003 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:33.752173 14003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:33.752748 14003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:33.753603 14003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:33.753652 14003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.753712 14003 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:33.753739 14003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.762367 14003 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:36645
I20250905 07:54:33.762392 14135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:36645 every 8 connection(s)
I20250905 07:54:33.763489 14003 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:33.766731 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14003
I20250905 07:54:33.767234 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:35395
--local_ip_for_outbound_sockets=127.4.49.3
--tserver_master_addrs=127.4.49.62:38767
--webserver_port=44267
--webserver_interface=127.4.49.3
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.49.3
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
W20250905 07:54:33.774484 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.765337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:33513 (local address 127.4.49.62:38767)
0905 07:54:33.765519 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.765524 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.766233 (+ 709us) server_negotiation.cc:408] Connection header received
0905 07:54:33.767246 (+ 1013us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.767250 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.767311 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.767434 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.769125 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.769844 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.770487 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.770682 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.774338 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:33513: BlockingRecv error: recv got EOF from 127.4.49.2:33513 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20250905 07:54:33.774962 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.765638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:33513)
0905 07:54:33.766088 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.766127 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.767010 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.767603 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.767615 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.768181 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.768941 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.768958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.769967 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.769969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.770367 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.770375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.770565 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.771600 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.771626 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.774134 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
W20250905 07:54:33.775519 14136 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:38767 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.49.62:38767: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.781834 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.776095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:41787)
0905 07:54:33.776328 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.776340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.776427 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.776675 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.776678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.776839 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.777036 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.777043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.778109 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.778112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.778513 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.778520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.778618 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.779155 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.779170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.781709 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20250905 07:54:33.782398 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.776197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:41787 (local address 127.4.49.62:38767)
0905 07:54:33.776337 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.776341 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.776382 (+ 41us) server_negotiation.cc:408] Connection header received
0905 07:54:33.776514 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.776518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.776571 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.776657 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.777152 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.777973 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.778622 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.778829 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.782249 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:41787: BlockingRecv error: recv got EOF from 127.4.49.2:41787 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.788185 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.782459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:36197)
0905 07:54:33.782662 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.782675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.782776 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.783106 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.783109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.783251 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.783440 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.783447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.784921 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.784925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.785322 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.785331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.785444 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.785998 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.786013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.788017 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20250905 07:54:33.788370 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.782564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:36197 (local address 127.4.49.62:38767)
0905 07:54:33.782701 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.782706 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.782728 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:54:33.782921 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.782925 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.782980 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.783067 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.783624 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.784490 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.785466 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.785697 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.788242 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:36197: BlockingRecv error: recv got EOF from 127.4.49.2:36197 (error 108)
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":71}
W20250905 07:54:33.788535 14136 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250905 07:54:33.884390 14139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:33.884656 14139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:33.884732 14139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:33.888200 14139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:33.888285 14139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:33.888386 14139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:33.893098 14139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/krb5kdc/oryx_127.4.49.3.keytab
--principal=oryx/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=44267
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:33.894248 14139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:33.895160 14139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:33.902335 14145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:33.902339 14144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.902400 14139 server_base.cc:1047] running on GCE node
W20250905 07:54:33.902339 14147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:33.902993 14139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:33.903648 14139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:33.904810 14139 hybrid_clock.cc:648] HybridClock initialized: now 1757058873904786 us; error 44 us; skew 500 ppm
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058873, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:33.907696 14139 init.cc:377] Logged in from keytab as oryx/127.4.49.3@KRBTEST.COM (short username oryx)
I20250905 07:54:33.908953 14139 webserver.cc:480] Webserver started at http://127.4.49.3:44267/ using document root <none> and password file <none>
I20250905 07:54:33.909575 14139 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:33.909646 14139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:33.913509 14139 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:33.915710 14154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.916888 14139 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:33.917018 14139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "d6c901aa80974660b7df2ba1cb2215d7"
format_stamp: "Formatted at 2025-09-05 07:54:33 on dist-test-slave-mnfb"
server_key: "b52c1e3021d91254ce397abae4f3752d"
server_key_iv: "52f5979409391bd1bb9f9d51fdab864b"
server_key_version: "encryptionkey@0"
I20250905 07:54:33.917493 14139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:33.929904 14139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:33.933440 14139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:33.933640 14139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:33.934254 14139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:33.935225 14139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:33.935276 14139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.935330 14139 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:33.935359 14139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:33.944657 14139 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:35395
I20250905 07:54:33.944679 14267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:35395 every 8 connection(s)
I20250905 07:54:33.945889 14139 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:33.954088 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14139
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
W20250905 07:54:33.957187 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.947950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:40421 (local address 127.4.49.62:38767)
0905 07:54:33.948130 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.948134 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.948949 (+ 815us) server_negotiation.cc:408] Connection header received
0905 07:54:33.949810 (+ 861us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.949813 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.949869 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.949945 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.951354 (+ 1409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.951880 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.952584 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.952747 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.957076 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:40421: BlockingRecv error: recv got EOF from 127.4.49.3:40421 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20250905 07:54:33.957791 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.948285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:40421)
0905 07:54:33.948775 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.948816 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.949579 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.950113 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.950123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.950555 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.951155 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.951170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.952005 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.952008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.952455 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.952464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.952655 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.953820 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.953850 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.956938 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
W20250905 07:54:33.958348 14268 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:38767 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.49.62:38767: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.963650 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.958856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:44973)
0905 07:54:33.958963 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.958977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.959055 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.959268 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.959271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.959425 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.959667 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.959676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.960491 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.960495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.961009 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.961020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.961138 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.961608 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.961625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.963510 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20250905 07:54:33.963773 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.958946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:44973 (local address 127.4.49.62:38767)
0905 07:54:33.959070 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.959075 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.959089 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:33.959126 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.959130 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.959174 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.959238 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.959821 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.960347 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.961149 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.961292 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.963674 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:44973: BlockingRecv error: recv got EOF from 127.4.49.3:44973 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:54:33.964112 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.955114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.4.49.62:38767)
0905 07:54:33.955391 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.955395 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.955407 (+ 12us) server_negotiation.cc:408] Connection header received
0905 07:54:33.955443 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.955446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.955508 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.955608 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.956689 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.957449 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.958209 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.958398 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.959579 (+ 1181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:33.959602 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:33.959621 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:33.959675 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:33.961920 (+ 2245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:33.962422 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:33.962430 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:33.962437 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:33.962542 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:33.962794 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:33.962798 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:33.962801 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:33.963147 (+ 346us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:33.963346 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:33.963659 (+ 313us) server_negotiation.cc:300] Negotiation successful
0905 07:54:33.963930 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:33 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:33.969604 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:33.964433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:57137)
0905 07:54:33.964554 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:33.964567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:33.964634 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:33.965381 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:33.965385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:33.965545 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:33.965776 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.965786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.966814 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.966817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:33.967208 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:33.967216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.967308 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.967795 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:33.967809 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:33.969487 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20250905 07:54:33.969787 14273 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:33.965059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:57137 (local address 127.4.49.62:38767)
0905 07:54:33.965194 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:33.965197 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:33.965207 (+ 10us) server_negotiation.cc:408] Connection header received
0905 07:54:33.965233 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:33.965236 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:33.965270 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:33.965329 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:33.965900 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.966686 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:33.967300 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:33.967450 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:33.969652 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:57137: BlockingRecv error: recv got EOF from 127.4.49.3:57137 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20250905 07:54:33.969853 14268 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:34.611563 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:34.606542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:51261)
0905 07:54:34.606840 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:34.606856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:34.606957 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:34.607239 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:34.607242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:34.607432 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:34.607643 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.607652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.608564 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.608567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:34.608992 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.608999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.609096 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.609696 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:34.609710 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:34.611415 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
W20250905 07:54:34.611704 14275 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:34.606656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:51261 (local address 127.4.49.62:38767)
0905 07:54:34.606956 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:34.606961 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:34.606979 (+ 18us) server_negotiation.cc:408] Connection header received
0905 07:54:34.607049 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:34.607053 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:34.607108 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:34.607189 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:34.607774 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.608440 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.609093 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.609274 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.611567 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:51261: BlockingRecv error: recv got EOF from 127.4.49.1:51261 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:34.794036 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:34.789275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:47597)
0905 07:54:34.789538 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:34.789551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:34.789661 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:34.789909 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:34.789912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:34.790084 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:34.790283 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.790290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.790995 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.790998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:34.791401 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.791408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.791507 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.792079 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:34.792097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:34.793895 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
W20250905 07:54:34.794202 14275 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:34.789359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:47597 (local address 127.4.49.62:38767)
0905 07:54:34.789523 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:34.789526 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:34.789615 (+ 89us) server_negotiation.cc:408] Connection header received
0905 07:54:34.789760 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:34.789763 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:34.789815 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:34.789878 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:34.790383 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.790901 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.791500 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.791638 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.794069 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:47597: BlockingRecv error: recv got EOF from 127.4.49.2:47597 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:34 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:34.975441 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:34.970693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:52227)
0905 07:54:34.970977 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:34.970991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:34.971101 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:34.971338 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:34.971340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:34.971519 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:34.971713 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.971734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.972451 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.972454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:34.972866 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:34.972874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.972968 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.973506 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:34.973520 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:34.975319 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20250905 07:54:34.975539 14275 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:34.970806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:52227 (local address 127.4.49.62:38767)
0905 07:54:34.970989 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:34.970993 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:34.971061 (+ 68us) server_negotiation.cc:408] Connection header received
0905 07:54:34.971182 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:34.971185 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:34.971236 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:34.971306 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:34.971832 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.972356 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:34.972970 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:34.973106 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:34.975442 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:52227: BlockingRecv error: recv got EOF from 127.4.49.3:52227 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:35.617494 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:35.612423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:55353)
0905 07:54:35.612770 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:35.612785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:35.612969 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:35.613236 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:35.613238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:35.613424 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:35.613650 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.613657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.614415 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.614418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:35.614824 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.614832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.614928 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.615519 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:35.615533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:35.617334 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
W20250905 07:54:35.617664 14279 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:35.612581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:55353 (local address 127.4.49.62:38767)
0905 07:54:35.612816 (+ 235us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:35.612820 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:35.612887 (+ 67us) server_negotiation.cc:408] Connection header received
0905 07:54:35.613068 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:35.613071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:35.613118 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:35.613196 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:35.613784 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.614309 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.614917 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.615051 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.617530 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:55353: BlockingRecv error: recv got EOF from 127.4.49.1:55353 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:35.800105 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:35.794978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:50261)
0905 07:54:35.795250 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:35.795264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:35.795374 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:35.795620 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:35.795623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:35.795816 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:35.796025 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.796033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.796900 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.796904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:35.797308 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.797317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.797464 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.798034 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:35.798050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:35.799913 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
W20250905 07:54:35.800240 14279 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:35.795111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:50261 (local address 127.4.49.62:38767)
0905 07:54:35.795249 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:35.795252 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:35.795330 (+ 78us) server_negotiation.cc:408] Connection header received
0905 07:54:35.795452 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:35.795455 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:35.795509 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:35.795586 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:35.796191 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.796701 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.797409 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.797565 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.800117 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:50261: BlockingRecv error: recv got EOF from 127.4.49.2:50261 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:35 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:35.981884 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:35.976382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:41707)
0905 07:54:35.976713 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:35.976727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:35.976826 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:35.977065 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:35.977067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:35.977244 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:35.977462 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.977470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.978309 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.978312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:35.978861 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:35.978874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.978990 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.979563 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:35.979578 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:35.981706 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20250905 07:54:35.981987 14279 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:35.976590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:41707 (local address 127.4.49.62:38767)
0905 07:54:35.976793 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:35.976798 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:35.976814 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:35.976914 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:35.976917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:35.976965 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:35.977036 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:35.977580 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.978174 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:35.978995 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:35.979181 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:35.981878 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:41707: BlockingRecv error: recv got EOF from 127.4.49.3:41707 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:36.623152 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:36.618420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:41891)
0905 07:54:36.618728 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:36.618742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:36.618837 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:36.619088 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:36.619090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:36.619272 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:36.619474 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.619481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.620234 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.620237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:36.620619 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.620626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.620729 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.621282 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:36.621296 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:36.623021 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20250905 07:54:36.623315 14283 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:36.618553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41891 (local address 127.4.49.62:38767)
0905 07:54:36.618810 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:36.618813 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:36.618829 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:36.618921 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:36.618924 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:36.618974 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:36.619065 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:36.619589 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.620143 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.620761 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.620917 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.623183 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:41891: BlockingRecv error: recv got EOF from 127.4.49.1:41891 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:36.806103 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:36.800982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:33043)
0905 07:54:36.801262 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:36.801276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:36.801406 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:36.801699 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:36.801703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:36.801898 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:36.802127 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.802137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.803075 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.803078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:36.803495 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.803503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.803605 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.804197 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:36.804211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:36.805972 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20250905 07:54:36.806236 14283 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:36.801072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:33043 (local address 127.4.49.62:38767)
0905 07:54:36.801226 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:36.801230 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:36.801363 (+ 133us) server_negotiation.cc:408] Connection header received
0905 07:54:36.801514 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:36.801517 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:36.801582 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:36.801656 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:36.802258 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.802969 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.803641 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.803804 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.806126 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:33043: BlockingRecv error: recv got EOF from 127.4.49.2:33043 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:36 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:36.987681 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:36.982675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:53675)
0905 07:54:36.982933 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:36.982945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:36.983046 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:36.983275 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:36.983278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:36.983489 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:36.983703 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.983711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.984613 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.984617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:36.985009 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:36.985018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.985114 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.985651 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:36.985666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:36.987512 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20250905 07:54:36.987825 14283 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:36.982773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:53675 (local address 127.4.49.62:38767)
0905 07:54:36.982908 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:36.982911 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:36.983004 (+ 93us) server_negotiation.cc:408] Connection header received
0905 07:54:36.983137 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:36.983140 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:36.983187 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:36.983249 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:36.983853 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.984502 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:36.985100 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:36.985238 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:36.987719 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:53675: BlockingRecv error: recv got EOF from 127.4.49.3:53675 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:37.635592 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:37.624195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:41295)
0905 07:54:37.624539 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:37.624555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:37.624752 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:37.625108 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:37.625111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:37.625313 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:37.625556 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.625565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.626396 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.626399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:37.626822 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.626830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.626936 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.632963 (+ 6027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:37.633002 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:37.635398 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
W20250905 07:54:37.635811 14287 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:37.624326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41295 (local address 127.4.49.62:38767)
0905 07:54:37.624672 (+ 346us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:37.624677 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:37.624697 (+ 20us) server_negotiation.cc:408] Connection header received
0905 07:54:37.624874 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:37.624879 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:37.624941 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:37.625044 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:37.625721 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.626260 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.626965 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.627129 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.635645 (+ 8516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:41295: BlockingRecv error: recv got EOF from 127.4.49.1:41295 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:37.813544 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:37.807118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:44615)
0905 07:54:37.807512 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:37.807529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:37.807666 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:37.808138 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:37.808142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:37.808547 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:37.808772 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.808781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.809593 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.809596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:37.810116 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.810130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.810280 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.811034 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:37.811049 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:37.813378 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
W20250905 07:54:37.813664 14287 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:37.807292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:44615 (local address 127.4.49.62:38767)
0905 07:54:37.807510 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:37.807514 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:37.807615 (+ 101us) server_negotiation.cc:408] Connection header received
0905 07:54:37.807935 (+ 320us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:37.807938 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:37.807995 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:37.808094 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:37.808932 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.809460 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.810282 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.810504 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.813561 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:44615: BlockingRecv error: recv got EOF from 127.4.49.2:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:37 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:37.993654 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:37.988592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:43289)
0905 07:54:37.988918 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:37.988935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:37.989048 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:37.989305 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:37.989308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:37.989516 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:37.989730 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.989737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.990519 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.990522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:37.990911 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:37.990918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.991025 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.991612 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:37.991627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:37.993487 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20250905 07:54:37.993783 14287 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:37.988821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:43289 (local address 127.4.49.62:38767)
0905 07:54:37.988965 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:37.988970 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:37.988991 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:54:37.989150 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:37.989152 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:37.989199 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:37.989264 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:37.989862 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.990367 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:37.991027 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:37.991205 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:37.993655 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:43289: BlockingRecv error: recv got EOF from 127.4.49.3:43289 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:38.641811 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:38.636467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:60291)
0905 07:54:38.636743 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:38.636756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:38.636845 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:38.637135 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:38.637138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:38.637324 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:38.637584 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.637592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.638398 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.638431 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:38.638857 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.638867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.638974 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:38.639643 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:38.639659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:38.641675 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
W20250905 07:54:38.642040 14291 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:38.636599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:60291 (local address 127.4.49.62:38767)
0905 07:54:38.636855 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:38.636859 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:38.636896 (+ 37us) server_negotiation.cc:408] Connection header received
0905 07:54:38.636934 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:38.636936 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:38.636985 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:38.637057 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:38.637720 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.638267 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.638995 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.639158 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:38.641874 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:60291: BlockingRecv error: recv got EOF from 127.4.49.1:60291 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:38.819610 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:38.814596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:51937)
0905 07:54:38.814919 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:38.814933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:38.815026 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:38.815313 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:38.815316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:38.815513 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:38.815719 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.815727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.816506 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.816509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:38.816916 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.816923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.817030 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:38.817621 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:38.817635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:38.819447 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
W20250905 07:54:38.819855 14291 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:38.814817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:51937 (local address 127.4.49.62:38767)
0905 07:54:38.814964 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:38.814969 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:38.814985 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:38.815144 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:38.815147 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:38.815197 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:38.815261 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:38.815851 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.816394 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.817032 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.817221 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:38.819610 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:51937: BlockingRecv error: recv got EOF from 127.4.49.2:51937 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:38 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:39.000126 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:38.994739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:43351)
0905 07:54:38.995125 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:38.995141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:38.995262 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:38.995596 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:38.995600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:38.995834 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:38.996124 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.996136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.996986 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.996990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:38.997410 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:38.997418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.997539 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:38.998108 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:38.998131 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:38.999955 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":89,"threads_started":1}
W20250905 07:54:39.000262 14291 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:38.994911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:43351 (local address 127.4.49.62:38767)
0905 07:54:38.995090 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:38.995094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:38.995230 (+ 136us) server_negotiation.cc:408] Connection header received
0905 07:54:38.995351 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:38.995354 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:38.995414 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:38.995509 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:38.996276 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.996865 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:38.997564 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:38.997729 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:39.000147 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:43351: BlockingRecv error: recv got EOF from 127.4.49.3:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 05 07:54:39 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:39 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:39.647444 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:39.642603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:39675)
0905 07:54:39.642912 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:39.642926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:39.643033 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:39.643275 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:39.643278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:39.643471 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:39.643672 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:39.643680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.644440 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:39.644443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:39.644850 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:39.644858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.644959 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:39.645579 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:39.645594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:39.647308 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
W20250905 07:54:39.647639 14295 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:39.642669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:39675 (local address 127.4.49.62:38767)
0905 07:54:39.642914 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:39.642918 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:39.642990 (+ 72us) server_negotiation.cc:408] Connection header received
0905 07:54:39.643112 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:39.643114 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:39.643166 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:39.643244 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:39.643792 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.644347 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:39.644943 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.645078 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:39.647503 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:39675: BlockingRecv error: recv got EOF from 127.4.49.1:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
Sep 05 07:54:39 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:39 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:39.830114 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:39.820451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:42037)
0905 07:54:39.820734 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:39.820752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:39.820879 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:39.821174 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:39.821177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:39.821375 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:39.821588 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:39.821596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.822411 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:39.822415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:39.822967 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:39.822980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.823106 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:39.827312 (+ 4206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:39.827334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:39.829914 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
W20250905 07:54:39.830253 14295 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:39.820527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:42037 (local address 127.4.49.62:38767)
0905 07:54:39.820670 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:39.820675 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:39.820830 (+ 155us) server_negotiation.cc:408] Connection header received
0905 07:54:39.820982 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:39.820986 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:39.821047 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:39.821132 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:39.821713 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.822304 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:39.823313 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:39.823482 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:39.830114 (+ 6632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:42037: BlockingRecv error: recv got EOF from 127.4.49.2:42037 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:40.005993 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:40.000910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:36941)
0905 07:54:40.001220 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:40.001237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:40.001357 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:40.001669 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:40.001672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:40.001867 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:40.002070 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.002077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.002795 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.002799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:40.003189 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.003197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.003293 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.003842 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:40.003855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:40.005824 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
W20250905 07:54:40.006125 14295 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:40.000976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:36941 (local address 127.4.49.62:38767)
0905 07:54:40.001138 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:40.001143 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:40.001319 (+ 176us) server_negotiation.cc:408] Connection header received
0905 07:54:40.001471 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:40.001474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:40.001525 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:40.001594 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:40.002170 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.002683 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.003276 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.003396 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.006018 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:36941: BlockingRecv error: recv got EOF from 127.4.49.3:36941 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:40.653321 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:40.648287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:35797)
0905 07:54:40.648633 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:40.648647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:40.648749 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:40.649064 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:40.649066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:40.649245 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:40.649448 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.649455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.650210 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.650212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:40.650605 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.650612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.650705 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.651250 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:40.651264 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:40.653190 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":78,"threads_started":1}
W20250905 07:54:40.653438 14299 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:40.648409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:35797 (local address 127.4.49.62:38767)
0905 07:54:40.648681 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:40.648685 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:40.648700 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:40.648891 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:40.648895 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:40.648949 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:40.649036 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:40.649594 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.650123 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.650692 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.650822 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.653313 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:35797: BlockingRecv error: recv got EOF from 127.4.49.1:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:40 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:40.836143 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:40.831135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:55657)
0905 07:54:40.831517 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:40.831531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:40.831642 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:40.831910 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:40.831913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:40.832158 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:40.832362 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.832370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.833112 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.833115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:40.833506 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:40.833514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.833623 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.834188 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:40.834201 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:40.835975 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":92,"threads_started":1}
W20250905 07:54:40.836299 14299 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:40.831193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:55657 (local address 127.4.49.62:38767)
0905 07:54:40.831344 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:40.831348 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:40.831612 (+ 264us) server_negotiation.cc:408] Connection header received
0905 07:54:40.831748 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:40.831751 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:40.831804 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:40.831869 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:40.832488 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.833005 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:40.833623 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:40.833818 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:40.836179 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:55657: BlockingRecv error: recv got EOF from 127.4.49.2:55657 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:41.012046 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:41.007018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:32871)
0905 07:54:41.007333 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:41.007349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:41.007487 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:41.007766 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:41.007769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:41.007970 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:41.008228 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.008235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.009024 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.009028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:41.009412 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.009419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.009532 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.010077 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:41.010092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:41.011883 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
W20250905 07:54:41.012181 14299 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:41.007151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:32871 (local address 127.4.49.62:38767)
0905 07:54:41.007332 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:41.007337 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:41.007439 (+ 102us) server_negotiation.cc:408] Connection header received
0905 07:54:41.007604 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:41.007607 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:41.007658 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:41.007727 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:41.008353 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.008921 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.009524 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.009705 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.012044 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:32871: BlockingRecv error: recv got EOF from 127.4.49.3:32871 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:41.659158 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:41.654164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:47453)
0905 07:54:41.654451 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:41.654465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:41.654565 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:41.654812 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:41.654815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:41.654989 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:41.655188 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.655195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.655930 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.655932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:41.656377 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.656387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.656493 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.657099 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:41.657114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:41.659021 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20250905 07:54:41.659338 14304 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:41.654353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:47453 (local address 127.4.49.62:38767)
0905 07:54:41.654595 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:41.654599 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:41.654612 (+ 13us) server_negotiation.cc:408] Connection header received
0905 07:54:41.654654 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:41.654658 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:41.654706 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:41.654788 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:41.655292 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.655834 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.656534 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.656701 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.659206 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:47453: BlockingRecv error: recv got EOF from 127.4.49.1:47453 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:41 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:41.842182 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:41.837034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:37429)
0905 07:54:41.837315 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:41.837327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:41.837455 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:41.837805 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:41.837808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:41.837985 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:41.838188 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.838195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.839122 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.839126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:41.839508 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:41.839516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.839618 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.840261 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:41.840275 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:41.842026 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
W20250905 07:54:41.842273 14304 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:41.837166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:37429 (local address 127.4.49.62:38767)
0905 07:54:41.837303 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:41.837306 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:41.837400 (+ 94us) server_negotiation.cc:408] Connection header received
0905 07:54:41.837587 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:41.837591 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:41.837648 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:41.837736 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:41.838304 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.838999 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:41.839642 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:41.839808 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:41.842175 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:37429: BlockingRecv error: recv got EOF from 127.4.49.2:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:42.018280 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:42.013089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:45281)
0905 07:54:42.013385 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:42.013399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:42.013575 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:42.013864 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:42.013867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:42.014065 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:42.014265 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.014273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.015014 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.015017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:42.015417 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.015424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.015538 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.016163 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:42.016177 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:42.018104 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
W20250905 07:54:42.018733 14304 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:42.013211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:45281 (local address 127.4.49.62:38767)
0905 07:54:42.013374 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:42.013379 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:42.013473 (+ 94us) server_negotiation.cc:408] Connection header received
0905 07:54:42.013687 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:42.013689 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:42.013751 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:42.013820 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:42.014395 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.014900 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.015531 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.015711 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.018512 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:45281: BlockingRecv error: recv got EOF from 127.4.49.3:45281 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:42.668545 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:42.660126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:51571)
0905 07:54:42.660426 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:42.660440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:42.660537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:42.660798 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:42.660800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:42.660977 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:42.661200 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.661208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.662089 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.662093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:42.662499 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.662507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.662625 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.666340 (+ 3715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:42.666360 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:42.668362 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
W20250905 07:54:42.668709 14308 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:42.660229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:51571 (local address 127.4.49.62:38767)
0905 07:54:42.660497 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:42.660501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:42.660517 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:42.660634 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:42.660637 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:42.660688 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:42.660769 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:42.661360 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.661941 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.662623 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.662810 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.668581 (+ 5771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:51571: BlockingRecv error: recv got EOF from 127.4.49.1:51571 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":57,"threads_started":1}
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:42 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:42.847997 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:42.843053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:60353)
0905 07:54:42.843351 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:42.843365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:42.843497 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:42.843772 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:42.843775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:42.843953 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:42.844198 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.844205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.844924 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.844927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:42.845314 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:42.845322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.845429 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.846004 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:42.846020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:42.847866 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":79,"threads_started":1}
W20250905 07:54:42.848277 14308 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:42.843163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:60353 (local address 127.4.49.62:38767)
0905 07:54:42.843320 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:42.843324 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:42.843433 (+ 109us) server_negotiation.cc:408] Connection header received
0905 07:54:42.843576 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:42.843579 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:42.843629 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:42.843706 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:42.844293 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.844814 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:42.845414 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:42.845568 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:42.848035 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:60353: BlockingRecv error: recv got EOF from 127.4.49.2:60353 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:43.024163 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:43.019200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:59883)
0905 07:54:43.019491 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:43.019507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:43.019637 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:43.019913 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:43.019916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:43.020154 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:43.020360 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.020367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.021061 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.021063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:43.021450 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.021456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.021551 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.022152 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:43.022166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:43.023989 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
W20250905 07:54:43.024286 14308 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:43.019264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:59883 (local address 127.4.49.62:38767)
0905 07:54:43.019406 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:43.019410 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:43.019596 (+ 186us) server_negotiation.cc:408] Connection header received
0905 07:54:43.019737 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:43.019740 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:43.019800 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:43.019867 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:43.020456 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.020971 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.021571 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.021714 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.024161 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:59883: BlockingRecv error: recv got EOF from 127.4.49.3:59883 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:43.674980 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:43.669566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:44749)
0905 07:54:43.669861 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:43.669880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:43.669994 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:43.670248 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:43.670250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:43.670471 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:43.670678 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.670686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.671422 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.671425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:43.671844 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.671853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.671956 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.672580 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:43.672595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:43.674805 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
W20250905 07:54:43.675096 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:43.669738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:44749 (local address 127.4.49.62:38767)
0905 07:54:43.670027 (+ 289us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:43.670031 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:43.670045 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:43.670086 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:43.670089 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:43.670136 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:43.670218 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:43.670797 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.671329 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.671941 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.672121 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.674981 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:44749: BlockingRecv error: recv got EOF from 127.4.49.1:44749 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:43 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:43.854233 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:43.848966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:60739)
0905 07:54:43.849316 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:43.849330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:43.849474 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:43.849763 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:43.849766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:43.849966 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:43.850164 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.850171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.850955 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.850958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:43.851347 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:43.851354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.851471 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.852090 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:43.852104 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:43.854073 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
W20250905 07:54:43.854360 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:43.849104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:60739 (local address 127.4.49.62:38767)
0905 07:54:43.849267 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:43.849271 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:43.849405 (+ 134us) server_negotiation.cc:408] Connection header received
0905 07:54:43.849574 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:43.849577 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:43.849629 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:43.849714 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:43.850318 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.850835 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:43.851465 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:43.851668 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:43.854241 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:60739: BlockingRecv error: recv got EOF from 127.4.49.2:60739 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:44.029783 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:44.025066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:54653)
0905 07:54:44.025383 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:44.025396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:44.025501 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:44.025725 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:44.025727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:44.025909 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:44.026107 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.026113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.026794 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.026797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:44.027174 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.027179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.027261 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.027831 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:44.027845 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:44.029638 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
W20250905 07:54:44.029963 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:44.025261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:54653 (local address 127.4.49.62:38767)
0905 07:54:44.025415 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:44.025419 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:44.025446 (+ 27us) server_negotiation.cc:408] Connection header received
0905 07:54:44.025580 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:44.025583 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:44.025627 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:44.025709 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:44.026205 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.026711 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.027251 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.027375 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.029833 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:54653: BlockingRecv error: recv got EOF from 127.4.49.3:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:44.681344 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:44.675914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:41287)
0905 07:54:44.676237 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:44.676254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:44.676371 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:44.676727 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:44.676731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:44.676918 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:44.677124 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.677131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.678108 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.678112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:44.678536 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.678546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.678689 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.679291 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:44.679306 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:44.681202 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
W20250905 07:54:44.681535 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:44.676081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:41287 (local address 127.4.49.62:38767)
0905 07:54:44.676315 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:44.676319 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:44.676334 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:44.676485 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:44.676488 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:44.676537 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:44.676621 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:44.677262 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.677829 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.678702 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.678877 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.681384 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:41287: BlockingRecv error: recv got EOF from 127.4.49.1:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:44 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:44.859735 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:44.854985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:36117)
0905 07:54:44.855257 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:44.855271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:44.855372 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:44.855589 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:44.855591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:44.855777 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:44.855982 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.855991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.856778 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.856781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:44.857167 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:44.857175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.857270 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.857863 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:44.857875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:44.859598 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
W20250905 07:54:44.859903 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:44.855051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:36117 (local address 127.4.49.62:38767)
0905 07:54:44.855201 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:44.855206 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:44.855329 (+ 123us) server_negotiation.cc:408] Connection header received
0905 07:54:44.855445 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:44.855447 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:44.855498 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:44.855562 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:44.856147 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.856655 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:44.857345 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:44.857527 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:44.859771 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:36117: BlockingRecv error: recv got EOF from 127.4.49.2:36117 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:45.035965 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:45.030837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:44313)
0905 07:54:45.031275 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:45.031290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:45.031414 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:45.031692 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:45.031696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:45.031886 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:45.032171 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.032181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.032921 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.032925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:45.033304 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.033311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.033414 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.033973 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:45.033987 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:45.035836 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":96,"threads_started":1}
W20250905 07:54:45.036149 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:45.030926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:44313 (local address 127.4.49.62:38767)
0905 07:54:45.031074 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:45.031079 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:45.031354 (+ 275us) server_negotiation.cc:408] Connection header received
0905 07:54:45.031513 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:45.031517 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:45.031576 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:45.031655 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:45.032300 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.032808 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.033414 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.033593 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.035991 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:44313: BlockingRecv error: recv got EOF from 127.4.49.3:44313 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:45.687525 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:45.682243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:47585)
0905 07:54:45.682563 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:45.682578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:45.682700 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:45.682972 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:45.682975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:45.683162 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:45.683369 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.683376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.684219 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.684223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:45.684637 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.684647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.684771 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.685374 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:45.685387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:45.687348 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
W20250905 07:54:45.687685 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:45.682413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:47585 (local address 127.4.49.62:38767)
0905 07:54:45.682646 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:45.682650 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:45.682666 (+ 16us) server_negotiation.cc:408] Connection header received
0905 07:54:45.682800 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:45.682803 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:45.682855 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:45.682940 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:45.683496 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.684035 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.684774 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.684924 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.687564 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:47585: BlockingRecv error: recv got EOF from 127.4.49.1:47585 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:45 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:45.865768 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:45.860602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:35615)
0905 07:54:45.860934 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:45.860948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:45.861163 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:45.861495 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:45.861498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:45.861703 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:45.861932 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.861941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.862692 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.862695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:45.863090 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:45.863097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.863188 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.863719 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:45.863733 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:45.865589 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
W20250905 07:54:45.865873 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:45.860717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:35615 (local address 127.4.49.62:38767)
0905 07:54:45.860919 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:45.860924 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:45.861023 (+ 99us) server_negotiation.cc:408] Connection header received
0905 07:54:45.861316 (+ 293us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:45.861319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:45.861371 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:45.861438 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:45.862049 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.862586 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:45.863195 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:45.863337 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:45.865768 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:35615: BlockingRecv error: recv got EOF from 127.4.49.2:35615 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:46.042295 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:46.036950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:35681)
0905 07:54:46.037249 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:46.037268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:46.037376 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:46.037763 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:46.037767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:46.037962 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:46.038172 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.038179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.038945 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.038948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:46.039336 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.039345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.039467 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.040123 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:46.040137 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:46.042137 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
W20250905 07:54:46.042429 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:46.037083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:35681 (local address 127.4.49.62:38767)
0905 07:54:46.037251 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:46.037255 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:46.037341 (+ 86us) server_negotiation.cc:408] Connection header received
0905 07:54:46.037530 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:46.037534 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:46.037603 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:46.037704 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:46.038308 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.038824 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.039454 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.039636 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.042307 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:35681: BlockingRecv error: recv got EOF from 127.4.49.3:35681 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:46.693794 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:46.688644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:54875)
0905 07:54:46.688934 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:46.688947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:46.689047 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:46.689320 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:46.689323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:46.689533 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:46.689805 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.689814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.690731 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.690734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:46.691121 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.691129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.691227 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.691792 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:46.691808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:46.693645 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
W20250905 07:54:46.694017 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:46.688831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:54875 (local address 127.4.49.62:38767)
0905 07:54:46.689082 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:46.689085 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:46.689099 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:46.689130 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:46.689133 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:46.689177 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:46.689277 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:46.689948 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.690600 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.691262 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.691419 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.693873 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:54875: BlockingRecv error: recv got EOF from 127.4.49.1:54875 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:46 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:46.871953 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:46.866812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:59161)
0905 07:54:46.867140 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:46.867157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:46.867288 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:46.867559 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:46.867562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:46.867780 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:46.867983 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.867990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.868843 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.868846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:46.869297 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:46.869308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.869422 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.870002 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:46.870016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:46.871822 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20250905 07:54:46.872128 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:46.867036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:59161 (local address 127.4.49.62:38767)
0905 07:54:46.867173 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:46.867179 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:46.867220 (+ 41us) server_negotiation.cc:408] Connection header received
0905 07:54:46.867387 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:46.867390 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:46.867442 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:46.867533 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:46.868124 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.868727 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:46.869429 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:46.869627 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:46.871976 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:59161: BlockingRecv error: recv got EOF from 127.4.49.2:59161 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:47.048818 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:47.043229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:41743)
0905 07:54:47.043601 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:47.043616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:47.043744 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:47.044011 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:47.044014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:47.044251 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:47.044461 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.044469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.045245 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.045249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:47.045643 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.045651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.045773 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.046462 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:47.046479 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:47.048654 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
W20250905 07:54:47.048913 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:47.043367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:41743 (local address 127.4.49.62:38767)
0905 07:54:47.043567 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:47.043572 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:47.043700 (+ 128us) server_negotiation.cc:408] Connection header received
0905 07:54:47.043842 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:47.043845 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:47.043897 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:47.043967 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:47.044609 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.045123 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.045764 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.045948 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.048814 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:41743: BlockingRecv error: recv got EOF from 127.4.49.3:41743 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:47.704389 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:47.694825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:48367)
0905 07:54:47.695247 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:47.695262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:47.695364 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:47.695778 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:47.695782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:47.695967 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:47.696222 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.696230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.697218 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.697223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:47.697754 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.697767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.697879 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.702206 (+ 4327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:47.702227 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:47.704213 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":107,"threads_started":1}
W20250905 07:54:47.704517 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:47.695007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:48367 (local address 127.4.49.62:38767)
0905 07:54:47.695295 (+ 288us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:47.695299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:47.695332 (+ 33us) server_negotiation.cc:408] Connection header received
0905 07:54:47.695516 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:47.695520 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:47.695576 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:47.695681 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:47.696382 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.697070 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.697900 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.698086 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.704398 (+ 6312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:48367: BlockingRecv error: recv got EOF from 127.4.49.1:48367 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":65,"threads_started":1}
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:47 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:47.877777 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:47.872809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:45905)
0905 07:54:47.873108 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:47.873124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:47.873249 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:47.873524 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:47.873527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:47.873718 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:47.873917 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.873924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.874680 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.874683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:47.875070 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:47.875078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.875186 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.875765 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:47.875781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:47.877627 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20250905 07:54:47.878012 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:47.872958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:45905 (local address 127.4.49.62:38767)
0905 07:54:47.873117 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:47.873123 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:47.873205 (+ 82us) server_negotiation.cc:408] Connection header received
0905 07:54:47.873344 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:47.873348 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:47.873400 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:47.873468 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:47.874060 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.874561 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:47.875187 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:47.875368 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:47.877777 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:45905: BlockingRecv error: recv got EOF from 127.4.49.2:45905 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.3@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:48.054615 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:48.049739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.3:48255)
0905 07:54:48.050057 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:48.050071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:48.050189 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:48.050438 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:48.050442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:48.050651 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:48.050856 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.050863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.051579 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.051582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:48.051969 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.051976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.052120 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.052752 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:48.052767 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:48.054459 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
W20250905 07:54:48.054738 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:48.049859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:48255 (local address 127.4.49.62:38767)
0905 07:54:48.050013 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:48.050018 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:48.050142 (+ 124us) server_negotiation.cc:408] Connection header received
0905 07:54:48.050280 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:48.050284 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:48.050332 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:48.050398 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:48.050979 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.051465 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.052105 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.052283 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.054623 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.3:48255: BlockingRecv error: recv got EOF from 127.4.49.3:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.1@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:48.710256 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:48.705492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.1:56309)
0905 07:54:48.705783 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:48.705798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:48.705900 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:48.706152 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:48.706154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:48.706335 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:48.706556 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.706564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.707284 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.707287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:48.707675 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.707682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.707770 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.708408 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:48.708421 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:48.710124 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20250905 07:54:48.710431 14332 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:48.705569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:56309 (local address 127.4.49.62:38767)
0905 07:54:48.705875 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:48.705879 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:48.705894 (+ 15us) server_negotiation.cc:408] Connection header received
0905 07:54:48.705988 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:48.705991 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:48.706045 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:48.706125 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:48.706661 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.707193 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.707766 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.707907 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.710297 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.1:56309: BlockingRecv error: recv got EOF from 127.4.49.1:56309 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 05 07:54:48 dist-test-slave-mnfb krb5kdc[13253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.49.2@KRBTEST.COM for oryx/127.4.49.62@KRBTEST.COM, Server not found in Kerberos database
I20250905 07:54:48.884166 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:48.878841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38767 (local address 127.4.49.2:42233)
0905 07:54:48.879185 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:48.879203 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:48.879326 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:48.879596 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:48.879600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:48.879817 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:48.880128 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.880139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.881015 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.881018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:48.881406 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:48.881414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.881541 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.882100 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:48.882115 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:48.883975 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.49.62:38767: Server oryx/127.4.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
W20250905 07:54:48.884363 14332 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:48.879020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:42233 (local address 127.4.49.62:38767)
0905 07:54:48.879194 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:48.879198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:48.879292 (+ 94us) server_negotiation.cc:408] Connection header received
0905 07:54:48.879421 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:48.879424 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:48.879478 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:48.879544 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:48.880261 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.880902 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:48.881550 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:48.881728 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:48.884225 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.49.2:42233: BlockingRecv error: recv got EOF from 127.4.49.2:42233 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20250905 07:54:48.962625 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13849
I20250905 07:54:48.967895 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14003
I20250905 07:54:48.972935 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14139
I20250905 07:54:48.978055 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 13779
2025-09-05T07:54:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18799 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-09-05T07:54:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:48Z Disabled control of system clock
I20250905 07:54:49.006309 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:35211
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:35211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250905 07:54:49.115759 14337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:49.116020 14337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:49.116097 14337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:49.119506 14337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:49.119589 14337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:49.119606 14337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:49.119624 14337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:49.119640 14337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:49.123765 14337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:35211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:35211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:49.125003 14337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:49.126214 14337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:49.132313 14345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.132380 14337 server_base.cc:1047] running on GCE node
W20250905 07:54:49.132306 14343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.132321 14342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.133100 14337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:49.134112 14337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:49.135288 14337 hybrid_clock.cc:648] HybridClock initialized: now 1757058889135271 us; error 36 us; skew 500 ppm
I20250905 07:54:49.137235 14337 webserver.cc:480] Webserver started at http://127.4.49.62:41977/ using document root <none> and password file <none>
I20250905 07:54:49.137863 14337 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:49.137923 14337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:49.138130 14337 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:49.139992 14337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "04b5612f20ba452cb0f8c8bea750f27b"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "9f3451a83c06539ff2c2a35eaecfcf09"
server_key_iv: "1c61a35fb852019f0db1e774d5660d48"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.140559 14337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "04b5612f20ba452cb0f8c8bea750f27b"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "9f3451a83c06539ff2c2a35eaecfcf09"
server_key_iv: "1c61a35fb852019f0db1e774d5660d48"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.144102 14337 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:54:49.146376 14351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.147399 14337 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:49.147519 14337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "04b5612f20ba452cb0f8c8bea750f27b"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "9f3451a83c06539ff2c2a35eaecfcf09"
server_key_iv: "1c61a35fb852019f0db1e774d5660d48"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.147627 14337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:49.160679 14337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:49.161430 14337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:49.161616 14337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:49.169019 14337 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:35211
I20250905 07:54:49.169014 14404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:35211 every 8 connection(s)
I20250905 07:54:49.170135 14337 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:49.171587 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14337
I20250905 07:54:49.171720 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:49.171981 4292 external_mini_cluster.cc:1468] Setting key b51e7b82162c79b5d8e8897484e5e523
I20250905 07:54:49.173408 14405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:49.180493 14405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b: Bootstrap starting.
I20250905 07:54:49.181062 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.173332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.4.49.62:35211)
0905 07:54:49.174040 (+ 708us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:49.174052 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:49.174095 (+ 43us) server_negotiation.cc:408] Connection header received
0905 07:54:49.174931 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:49.174959 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:49.175416 (+ 457us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:49.175865 (+ 449us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:49.176460 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.177671 (+ 1211us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.178315 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.178605 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.178994 (+ 389us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:49.179047 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:49.179062 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:49.179572 (+ 510us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:49.179627 (+ 55us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:49.179646 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:49.179749 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:49.179868 (+ 119us) server_negotiation.cc:300] Negotiation successful
0905 07:54:49.180167 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20250905 07:54:49.183197 14405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:49.183983 14405 log.cc:826] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:49.186003 14405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b: No bootstrap required, opened a new log
I20250905 07:54:49.188788 14405 raft_consensus.cc:357] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } }
I20250905 07:54:49.189008 14405 raft_consensus.cc:383] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:49.189054 14405 raft_consensus.cc:738] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b5612f20ba452cb0f8c8bea750f27b, State: Initialized, Role: FOLLOWER
I20250905 07:54:49.189448 14405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } }
I20250905 07:54:49.189603 14405 raft_consensus.cc:397] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:49.189677 14405 raft_consensus.cc:491] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:49.189798 14405 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:49.190914 14405 raft_consensus.cc:513] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } }
I20250905 07:54:49.191305 14405 leader_election.cc:304] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b5612f20ba452cb0f8c8bea750f27b; no voters:
I20250905 07:54:49.191612 14405 leader_election.cc:290] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:49.191735 14410 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:49.191916 14410 raft_consensus.cc:695] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [term 1 LEADER]: Becoming Leader. State: Replica: 04b5612f20ba452cb0f8c8bea750f27b, State: Running, Role: LEADER
I20250905 07:54:49.192294 14410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } }
I20250905 07:54:49.192811 14405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:49.193884 14412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b5612f20ba452cb0f8c8bea750f27b. Latest consensus state: current_term: 1 leader_uuid: "04b5612f20ba452cb0f8c8bea750f27b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } } }
I20250905 07:54:49.194020 14412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:49.194397 14420 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:49.194440 14411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b5612f20ba452cb0f8c8bea750f27b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b5612f20ba452cb0f8c8bea750f27b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 35211 } } }
I20250905 07:54:49.194527 14411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:49.197172 14420 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:49.202571 14420 catalog_manager.cc:1349] Generated new cluster ID: 6391247d6b9a476e952d8d4afb892064
I20250905 07:54:49.202666 14420 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:49.222185 14420 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:49.223225 14420 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:49.228696 14420 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 04b5612f20ba452cb0f8c8bea750f27b: Generated new TSK 0
I20250905 07:54:49.229467 14420 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:54:49.235607 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--builtin_ntp_servers=127.4.49.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20250905 07:54:49.350132 14430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:49.350391 14430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:49.350459 14430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:49.354056 14430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:49.354136 14430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:49.354238 14430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:49.358316 14430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:49.359479 14430 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:49.360684 14430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:49.369408 14437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.369408 14436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.369408 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.369796 14430 server_base.cc:1047] running on GCE node
I20250905 07:54:49.370231 14430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:49.370855 14430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:49.372021 14430 hybrid_clock.cc:648] HybridClock initialized: now 1757058889371998 us; error 39 us; skew 500 ppm
I20250905 07:54:49.373947 14430 webserver.cc:480] Webserver started at http://127.4.49.1:36709/ using document root <none> and password file <none>
I20250905 07:54:49.374531 14430 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:49.374590 14430 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:49.374802 14430 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:49.376677 14430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "f9f0c0e917bc463384fd4bbcb8dc9bce"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "25494b80042923f7da50a0d617c5102a"
server_key_iv: "903716caee300ab0ae6372cac2e5d804"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.377174 14430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "f9f0c0e917bc463384fd4bbcb8dc9bce"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "25494b80042923f7da50a0d617c5102a"
server_key_iv: "903716caee300ab0ae6372cac2e5d804"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.380779 14430 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:54:49.383227 14445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.384388 14430 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:49.384516 14430 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "f9f0c0e917bc463384fd4bbcb8dc9bce"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "25494b80042923f7da50a0d617c5102a"
server_key_iv: "903716caee300ab0ae6372cac2e5d804"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.384625 14430 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:49.407234 14430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:49.407955 14430 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:49.408164 14430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:49.408754 14430 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:49.409797 14430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:49.409848 14430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.409906 14430 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:49.409933 14430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.419504 14430 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:44363
I20250905 07:54:49.419535 14560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:44363 every 8 connection(s)
I20250905 07:54:49.420583 14430 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:49.422396 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14430
I20250905 07:54:49.422515 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:49.422797 4292 external_mini_cluster.cc:1468] Setting key 0f6361aa2e0309ddf07a8afc3def3a00
I20250905 07:54:49.424679 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--builtin_ntp_servers=127.4.49.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250905 07:54:49.429391 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.422601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:43411 (local address 127.4.49.62:35211)
0905 07:54:49.422765 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:49.422769 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:49.423790 (+ 1021us) server_negotiation.cc:408] Connection header received
0905 07:54:49.424698 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:49.424702 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:49.424769 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:49.424838 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:49.425921 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.426649 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.427282 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.427471 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.428818 (+ 1347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:49.428846 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:49.428850 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:49.428870 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:49.428898 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:49.428908 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:49.428994 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:49.429130 (+ 136us) server_negotiation.cc:300] Negotiation successful
0905 07:54:49.429220 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250905 07:54:49.430065 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.422897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35211 (local address 127.4.49.1:43411)
0905 07:54:49.423657 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:49.423703 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:49.424496 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:49.425092 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:49.425101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:49.425185 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:49.425785 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.425798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.426776 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.426778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:49.427179 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.427185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.427354 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.428551 (+ 1197us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:49.428553 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:49.428593 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:49.428661 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:49.428994 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:49.428999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:49.429003 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:54:49.429114 (+ 111us) client_negotiation.cc:240] Negotiation successful
0905 07:54:49.429339 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20250905 07:54:49.431244 14561 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35211
I20250905 07:54:49.431555 14561 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:49.432403 14561 heartbeater.cc:507] Master 127.4.49.62:35211 requested a full tablet report, sending...
I20250905 07:54:49.434180 14368 ts_manager.cc:194] Registered new tserver with Master: f9f0c0e917bc463384fd4bbcb8dc9bce (127.4.49.1:44363)
I20250905 07:54:49.435427 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.49.1:43411
W20250905 07:54:49.535708 14564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:49.535960 14564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:49.536010 14564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:49.539491 14564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:49.539570 14564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:49.539661 14564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:49.543679 14564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:49.544901 14564 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:49.546085 14564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:49.552726 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.552754 14569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.552727 14570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.553125 14564 server_base.cc:1047] running on GCE node
I20250905 07:54:49.553656 14564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:49.554360 14564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:49.555567 14564 hybrid_clock.cc:648] HybridClock initialized: now 1757058889555547 us; error 39 us; skew 500 ppm
I20250905 07:54:49.557595 14564 webserver.cc:480] Webserver started at http://127.4.49.2:36655/ using document root <none> and password file <none>
I20250905 07:54:49.558197 14564 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:49.558261 14564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:49.558459 14564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:49.560387 14564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "2a67f75697dc4c649942dbe9b11f2e80"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5dd0f15abf17c6026e024260c2922179"
server_key_iv: "c73b90acf281168b2aa7f5c469aa1b2d"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.560889 14564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "2a67f75697dc4c649942dbe9b11f2e80"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5dd0f15abf17c6026e024260c2922179"
server_key_iv: "c73b90acf281168b2aa7f5c469aa1b2d"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.564420 14564 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:54:49.566887 14579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.568147 14564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:49.568295 14564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "2a67f75697dc4c649942dbe9b11f2e80"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5dd0f15abf17c6026e024260c2922179"
server_key_iv: "c73b90acf281168b2aa7f5c469aa1b2d"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.568408 14564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:49.578801 14564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:49.579520 14564 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:49.579701 14564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:49.580371 14564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:49.581430 14564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:49.581485 14564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.581540 14564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:49.581562 14564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.590538 14564 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:45207
I20250905 07:54:49.590595 14693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:45207 every 8 connection(s)
I20250905 07:54:49.591603 14564 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:49.600083 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.593463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:40119 (local address 127.4.49.62:35211)
0905 07:54:49.593601 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:49.593605 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:49.594689 (+ 1084us) server_negotiation.cc:408] Connection header received
0905 07:54:49.595619 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:49.595622 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:49.595684 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:49.595768 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:49.596877 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.597453 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.598068 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.598197 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.599500 (+ 1303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:49.599523 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:49.599526 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:49.599543 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:49.599567 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:49.599575 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:49.599637 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:49.599850 (+ 213us) server_negotiation.cc:300] Negotiation successful
0905 07:54:49.599922 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:49.600455 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14564
I20250905 07:54:49.600570 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:49.600878 4292 external_mini_cluster.cc:1468] Setting key 77fadb70953dec284428684ae8b80b53
I20250905 07:54:49.601058 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.593814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35211 (local address 127.4.49.2:40119)
0905 07:54:49.594538 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:49.594581 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:49.595410 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:49.595931 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:49.595939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:49.596026 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:49.596701 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.596716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.597573 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.597576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:49.597982 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.597989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.598167 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.599244 (+ 1077us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:49.599246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:49.599283 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:49.599352 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:49.599690 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:49.599699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:49.599704 (+ 5us) client_negotiation.cc:769] Sending connection context
0905 07:54:49.599856 (+ 152us) client_negotiation.cc:240] Negotiation successful
0905 07:54:49.600131 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20250905 07:54:49.602231 14694 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35211
I20250905 07:54:49.602540 14694 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:49.602926 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--builtin_ntp_servers=127.4.49.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250905 07:54:49.603163 14694 heartbeater.cc:507] Master 127.4.49.62:35211 requested a full tablet report, sending...
I20250905 07:54:49.604333 14368 ts_manager.cc:194] Registered new tserver with Master: 2a67f75697dc4c649942dbe9b11f2e80 (127.4.49.2:45207)
I20250905 07:54:49.604923 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.49.2:40119
W20250905 07:54:49.717177 14697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:49.717463 14697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:49.717545 14697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:49.720995 14697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:49.721069 14697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:49.721161 14697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:49.725199 14697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:35211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:49.726353 14697 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:49.727552 14697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:49.734597 14705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.734725 14697 server_base.cc:1047] running on GCE node
W20250905 07:54:49.734586 14703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.734587 14702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.735266 14697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:49.735841 14697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:49.736999 14697 hybrid_clock.cc:648] HybridClock initialized: now 1757058889736976 us; error 39 us; skew 500 ppm
I20250905 07:54:49.739075 14697 webserver.cc:480] Webserver started at http://127.4.49.3:45623/ using document root <none> and password file <none>
I20250905 07:54:49.739660 14697 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:49.739713 14697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:49.739900 14697 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:49.741784 14697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "95082470772546218fd1b509e86c8488"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "566f8a42b054ee27728e5ab48e790a34"
server_key_iv: "7c4ca6af57271e9a035a191c6a32c7c9"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.742266 14697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "95082470772546218fd1b509e86c8488"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "566f8a42b054ee27728e5ab48e790a34"
server_key_iv: "7c4ca6af57271e9a035a191c6a32c7c9"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.745872 14697 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:49.748376 14711 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.749485 14697 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 07:54:49.749606 14697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "95082470772546218fd1b509e86c8488"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "566f8a42b054ee27728e5ab48e790a34"
server_key_iv: "7c4ca6af57271e9a035a191c6a32c7c9"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.749702 14697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:49.764536 14697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:49.765237 14697 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:49.765394 14697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:49.765931 14697 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:49.766953 14697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:49.767005 14697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.767048 14697 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:49.767068 14697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.776381 14697 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:35979
I20250905 07:54:49.776413 14826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:35979 every 8 connection(s)
I20250905 07:54:49.777503 14697 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:49.779081 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14697
I20250905 07:54:49.779196 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:49.779464 4292 external_mini_cluster.cc:1468] Setting key 7c45a0689a7ec40d58a4709ea453201e
I20250905 07:54:49.786190 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.779444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:59845 (local address 127.4.49.62:35211)
0905 07:54:49.779585 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:49.779589 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:49.780862 (+ 1273us) server_negotiation.cc:408] Connection header received
0905 07:54:49.781791 (+ 929us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:49.781795 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:49.781853 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:49.781920 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:49.782954 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.783457 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.784160 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.784342 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.785657 (+ 1315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:49.785681 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:49.785684 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:49.785699 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:49.785721 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:49.785728 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:49.785789 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:49.785952 (+ 163us) server_negotiation.cc:300] Negotiation successful
0905 07:54:49.786030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:54:49.786912 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:49.779906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:35211 (local address 127.4.49.3:59845)
0905 07:54:49.780714 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:49.780774 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:49.781590 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:49.782085 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:49.782094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:49.782183 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:49.782809 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.782823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.783589 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:49.783592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:49.784010 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:49.784017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:49.784278 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:49.785403 (+ 1125us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:49.785406 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:49.785444 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:49.785514 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:49.785803 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:49.785812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:49.785818 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:54:49.785953 (+ 135us) client_negotiation.cc:240] Negotiation successful
0905 07:54:49.786228 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":165,"threads_started":1}
I20250905 07:54:49.787997 14827 heartbeater.cc:344] Connected to a master server at 127.4.49.62:35211
I20250905 07:54:49.788285 14827 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:49.788913 14827 heartbeater.cc:507] Master 127.4.49.62:35211 requested a full tablet report, sending...
I20250905 07:54:49.790046 14368 ts_manager.cc:194] Registered new tserver with Master: 95082470772546218fd1b509e86c8488 (127.4.49.3:35979)
I20250905 07:54:49.790622 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.49.3:59845
I20250905 07:54:49.793416 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250905 07:54:49.796171 14408 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:49.795266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.4.49.62:35211)
0905 07:54:49.795382 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:49.795385 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:49.795591 (+ 206us) server_negotiation.cc:408] Connection header received
0905 07:54:49.795698 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:49.795701 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:49.795758 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:49.795826 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:49.796020 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51394: BlockingRecv error: recv got EOF from 127.0.0.1:51394 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20250905 07:54:49.796346 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14430
I20250905 07:54:49.803099 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14564
I20250905 07:54:49.808171 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14697
I20250905 07:54:49.813410 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14337
2025-09-05T07:54:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (834 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-09-05T07:54:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:49Z Disabled control of system clock
I20250905 07:54:49.841526 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36209
--webserver_interface=127.4.49.62
--webserver_port=0
--builtin_ntp_servers=127.4.49.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:36209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250905 07:54:49.968143 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:49.968397 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:49.968456 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:49.971832 14839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:49.971920 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:49.971936 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:49.971956 14839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:49.971972 14839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:49.976122 14839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:36209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:49.977288 14839 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:49.978463 14839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:49.984166 14845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.984205 14839 server_base.cc:1047] running on GCE node
W20250905 07:54:49.984220 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:49.984166 14844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:49.984977 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:49.985947 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:49.987113 14839 hybrid_clock.cc:648] HybridClock initialized: now 1757058889987090 us; error 44 us; skew 500 ppm
I20250905 07:54:49.989017 14839 webserver.cc:480] Webserver started at http://127.4.49.62:35427/ using document root <none> and password file <none>
I20250905 07:54:49.989614 14839 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:49.989676 14839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:49.989871 14839 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:49.991712 14839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "27d63e1117d24f9999cf247de7d5c416"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5e3e7e4c8f6fca6970ad76f43c9d8e34"
server_key_iv: "92dff6b68bc8d9b1d6055e4db6cefea7"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.992260 14839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "27d63e1117d24f9999cf247de7d5c416"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5e3e7e4c8f6fca6970ad76f43c9d8e34"
server_key_iv: "92dff6b68bc8d9b1d6055e4db6cefea7"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.995818 14839 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:54:49.998178 14853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:49.999235 14839 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:54:49.999353 14839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "27d63e1117d24f9999cf247de7d5c416"
format_stamp: "Formatted at 2025-09-05 07:54:49 on dist-test-slave-mnfb"
server_key: "5e3e7e4c8f6fca6970ad76f43c9d8e34"
server_key_iv: "92dff6b68bc8d9b1d6055e4db6cefea7"
server_key_version: "encryptionkey@0"
I20250905 07:54:49.999449 14839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:50.004360 14839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:50.005096 14839 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:50.005275 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:50.012565 14906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:36209 every 8 connection(s)
I20250905 07:54:50.012570 14839 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:36209
I20250905 07:54:50.013657 14839 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:50.016551 14907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:50.016976 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14839
I20250905 07:54:50.017110 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:50.017381 4292 external_mini_cluster.cc:1468] Setting key 74145466a545e0435a875cde16b7a41e
I20250905 07:54:50.022562 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.018737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.4.49.62:36209)
0905 07:54:50.019378 (+ 641us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:50.019387 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:50.019422 (+ 35us) server_negotiation.cc:408] Connection header received
0905 07:54:50.019970 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:50.019988 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:50.020367 (+ 379us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:50.020688 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:50.021478 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:50.021483 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:50.021493 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:50.021504 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:50.021545 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:50.021557 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:50.021688 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:50.021785 (+ 97us) server_negotiation.cc:300] Negotiation successful
0905 07:54:50.022001 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20250905 07:54:50.023006 14907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416: Bootstrap starting.
I20250905 07:54:50.025291 14907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:50.026027 14907 log.cc:826] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:50.027973 14907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416: No bootstrap required, opened a new log
I20250905 07:54:50.030673 14907 raft_consensus.cc:357] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } }
I20250905 07:54:50.030918 14907 raft_consensus.cc:383] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:50.030972 14907 raft_consensus.cc:738] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d63e1117d24f9999cf247de7d5c416, State: Initialized, Role: FOLLOWER
I20250905 07:54:50.031453 14907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } }
I20250905 07:54:50.031626 14907 raft_consensus.cc:397] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:50.031700 14907 raft_consensus.cc:491] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:50.031818 14907 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:50.032974 14907 raft_consensus.cc:513] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } }
I20250905 07:54:50.033341 14907 leader_election.cc:304] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27d63e1117d24f9999cf247de7d5c416; no voters:
I20250905 07:54:50.033650 14907 leader_election.cc:290] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:50.033823 14912 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:50.034034 14912 raft_consensus.cc:695] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [term 1 LEADER]: Becoming Leader. State: Replica: 27d63e1117d24f9999cf247de7d5c416, State: Running, Role: LEADER
I20250905 07:54:50.034355 14912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } }
I20250905 07:54:50.034890 14907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:50.036463 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27d63e1117d24f9999cf247de7d5c416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } } }
I20250905 07:54:50.036623 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:50.037097 14921 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:50.037204 14914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d63e1117d24f9999cf247de7d5c416. Latest consensus state: current_term: 1 leader_uuid: "27d63e1117d24f9999cf247de7d5c416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d63e1117d24f9999cf247de7d5c416" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 36209 } } }
I20250905 07:54:50.037590 14914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416 [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:50.040522 14921 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:50.046137 14921 catalog_manager.cc:1349] Generated new cluster ID: cfda4a9f36964bd581152d72adc68b35
I20250905 07:54:50.046237 14921 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:50.065706 14921 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:50.066056 14921 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:50.081166 14921 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 27d63e1117d24f9999cf247de7d5c416: Generated new TSK 0
I20250905 07:54:50.081974 14921 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:54:50.090939 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--builtin_ntp_servers=127.4.49.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250905 07:54:50.204520 14931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:50.204774 14931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:50.204849 14931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:50.208304 14931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:50.208381 14931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:50.208495 14931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:50.212563 14931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:50.213793 14931 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:50.215054 14931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:50.222303 14936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:50.222482 14931 server_base.cc:1047] running on GCE node
W20250905 07:54:50.222303 14939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:50.222326 14937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:50.223066 14931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:50.223712 14931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:50.224874 14931 hybrid_clock.cc:648] HybridClock initialized: now 1757058890224852 us; error 43 us; skew 500 ppm
I20250905 07:54:50.226926 14931 webserver.cc:480] Webserver started at http://127.4.49.1:38425/ using document root <none> and password file <none>
I20250905 07:54:50.227538 14931 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:50.227603 14931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:50.227800 14931 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:50.229688 14931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "0a8b0e261e8544f895b4ba473b09b2e3"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "f85034b1d2b065af030ad00fbc6a9386"
server_key_iv: "0623e7daeedb6e838ecc48a4ea5eb065"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.230196 14931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "0a8b0e261e8544f895b4ba473b09b2e3"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "f85034b1d2b065af030ad00fbc6a9386"
server_key_iv: "0623e7daeedb6e838ecc48a4ea5eb065"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.233932 14931 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:50.236539 14945 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.237829 14931 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:54:50.237968 14931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "0a8b0e261e8544f895b4ba473b09b2e3"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "f85034b1d2b065af030ad00fbc6a9386"
server_key_iv: "0623e7daeedb6e838ecc48a4ea5eb065"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.238082 14931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:50.243454 14931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:50.244200 14931 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:50.244469 14931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:50.245110 14931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:50.246145 14931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:50.246197 14931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.246251 14931 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:50.246280 14931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.255293 14931 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:33565
I20250905 07:54:50.255322 15059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:33565 every 8 connection(s)
I20250905 07:54:50.256371 14931 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:50.256661 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 14931
I20250905 07:54:50.256767 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:50.257035 4292 external_mini_cluster.cc:1468] Setting key d27a1e9bf89a4f852920fa259640b9ac
I20250905 07:54:50.259339 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--builtin_ntp_servers=127.4.49.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250905 07:54:50.262742 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.258453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:45751 (local address 127.4.49.62:36209)
0905 07:54:50.258588 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:50.258592 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:50.259954 (+ 1362us) server_negotiation.cc:408] Connection header received
0905 07:54:50.260933 (+ 979us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:50.260937 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:50.260977 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:50.261049 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:50.262253 (+ 1204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:50.262256 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:50.262258 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:50.262272 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:50.262294 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:50.262302 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:50.262379 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:50.262564 (+ 185us) server_negotiation.cc:300] Negotiation successful
0905 07:54:50.262608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250905 07:54:50.263816 15062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.258957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36209 (local address 127.4.49.1:45751)
0905 07:54:50.259786 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:50.259838 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:50.260691 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:50.261234 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:50.261245 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:50.261356 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:50.261964 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:50.261966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:50.262014 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:50.262089 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:50.262385 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:50.262394 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:50.262400 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:54:50.262619 (+ 219us) client_negotiation.cc:240] Negotiation successful
0905 07:54:50.262921 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":124,"threads_started":1}
I20250905 07:54:50.265408 15060 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36209
I20250905 07:54:50.265662 15060 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:50.266130 15060 heartbeater.cc:507] Master 127.4.49.62:36209 requested a full tablet report, sending...
I20250905 07:54:50.267858 14870 ts_manager.cc:194] Registered new tserver with Master: 0a8b0e261e8544f895b4ba473b09b2e3 (127.4.49.1:33565)
W20250905 07:54:50.370441 15063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:50.370697 15063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:50.370751 15063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:50.374099 15063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:50.374173 15063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:50.374251 15063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:50.378400 15063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:50.379523 15063 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:50.380745 15063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:50.387643 15069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:50.387631 15068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:50.387631 15071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:50.387866 15063 server_base.cc:1047] running on GCE node
I20250905 07:54:50.388406 15063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:50.389019 15063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:50.390182 15063 hybrid_clock.cc:648] HybridClock initialized: now 1757058890390164 us; error 32 us; skew 500 ppm
I20250905 07:54:50.392170 15063 webserver.cc:480] Webserver started at http://127.4.49.2:40161/ using document root <none> and password file <none>
I20250905 07:54:50.392781 15063 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:50.392850 15063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:50.393049 15063 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:50.394876 15063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "3c016daf95414471b7c9e9e5d91fdb86"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "e87a92c1c01ae7f5bd5e8cc872c641aa"
server_key_iv: "da9ebdfdded78e360217e40fd3410ede"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.395380 15063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "3c016daf95414471b7c9e9e5d91fdb86"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "e87a92c1c01ae7f5bd5e8cc872c641aa"
server_key_iv: "da9ebdfdded78e360217e40fd3410ede"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.398986 15063 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20250905 07:54:50.401648 15077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.402767 15063 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:54:50.402889 15063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "3c016daf95414471b7c9e9e5d91fdb86"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "e87a92c1c01ae7f5bd5e8cc872c641aa"
server_key_iv: "da9ebdfdded78e360217e40fd3410ede"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.402992 15063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:50.408152 15063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:50.408815 15063 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:50.409096 15063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:50.409703 15063 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:50.410723 15063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:50.410776 15063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.410830 15063 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:50.410858 15063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.419821 15063 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:38133
I20250905 07:54:50.419847 15191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:38133 every 8 connection(s)
I20250905 07:54:50.420883 15063 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:50.425778 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15063
I20250905 07:54:50.425896 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:50.426175 4292 external_mini_cluster.cc:1468] Setting key c250b8ebea30cddf9774a6e258ec6b80
I20250905 07:54:50.426667 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.422780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:60563 (local address 127.4.49.62:36209)
0905 07:54:50.422918 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:50.422921 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:50.423868 (+ 947us) server_negotiation.cc:408] Connection header received
0905 07:54:50.424749 (+ 881us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:50.424752 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:50.424807 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:50.424882 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:50.426103 (+ 1221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:50.426107 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:50.426110 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:50.426129 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:50.426157 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:50.426168 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:50.426287 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:50.426485 (+ 198us) server_negotiation.cc:300] Negotiation successful
0905 07:54:50.426539 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250905 07:54:50.427529 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.423030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36209 (local address 127.4.49.2:60563)
0905 07:54:50.423703 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:50.423754 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:50.424533 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:50.425059 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:50.425067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:50.425161 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:50.425758 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:50.425760 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:50.425794 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:50.425857 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:50.426304 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:50.426314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:50.426320 (+ 6us) client_negotiation.cc:769] Sending connection context
0905 07:54:50.426493 (+ 173us) client_negotiation.cc:240] Negotiation successful
0905 07:54:50.426757 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20250905 07:54:50.428359 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--builtin_ntp_servers=127.4.49.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250905 07:54:50.428886 15192 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36209
I20250905 07:54:50.429136 15192 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:50.429656 15192 heartbeater.cc:507] Master 127.4.49.62:36209 requested a full tablet report, sending...
I20250905 07:54:50.430807 14870 ts_manager.cc:194] Registered new tserver with Master: 3c016daf95414471b7c9e9e5d91fdb86 (127.4.49.2:38133)
W20250905 07:54:50.537725 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:50.537976 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:50.538039 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:50.541477 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:50.541548 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:50.541639 15195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:50.545717 15195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:50.546883 15195 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:50.548118 15195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:50.554991 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:50.554991 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:50.555022 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:50.555233 15195 server_base.cc:1047] running on GCE node
I20250905 07:54:50.555600 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:50.556231 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:50.557386 15195 hybrid_clock.cc:648] HybridClock initialized: now 1757058890557375 us; error 33 us; skew 500 ppm
I20250905 07:54:50.559274 15195 webserver.cc:480] Webserver started at http://127.4.49.3:36039/ using document root <none> and password file <none>
I20250905 07:54:50.559862 15195 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:50.559924 15195 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:50.560153 15195 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:50.561961 15195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "df00a0ee826d46e98b15c515d182a108"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "1cfe52faba5faaf227a8551edb8f191a"
server_key_iv: "d78e179b6bd07c14e826ef7dde2bb055"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.562443 15195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "df00a0ee826d46e98b15c515d182a108"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "1cfe52faba5faaf227a8551edb8f191a"
server_key_iv: "d78e179b6bd07c14e826ef7dde2bb055"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.565986 15195 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 07:54:50.568423 15209 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.569515 15195 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:50.569630 15195 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "df00a0ee826d46e98b15c515d182a108"
format_stamp: "Formatted at 2025-09-05 07:54:50 on dist-test-slave-mnfb"
server_key: "1cfe52faba5faaf227a8551edb8f191a"
server_key_iv: "d78e179b6bd07c14e826ef7dde2bb055"
server_key_version: "encryptionkey@0"
I20250905 07:54:50.569727 15195 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:50.574625 15195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:50.575268 15195 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:50.575424 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:50.575958 15195 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:50.576952 15195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:50.577000 15195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.577051 15195 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:50.577077 15195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:50.586324 15195 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:41885
I20250905 07:54:50.586380 15323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:41885 every 8 connection(s)
I20250905 07:54:50.587312 15195 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:50.592871 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.589167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:51251 (local address 127.4.49.62:36209)
0905 07:54:50.589317 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:50.589321 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:50.590417 (+ 1096us) server_negotiation.cc:408] Connection header received
0905 07:54:50.591212 (+ 795us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:50.591215 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:50.591265 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:50.591347 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:50.592476 (+ 1129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:50.592479 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:50.592482 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0905 07:54:50.592493 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:50.592511 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0905 07:54:50.592518 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:50.592596 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:50.592702 (+ 106us) server_negotiation.cc:300] Negotiation successful
0905 07:54:50.592740 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:54:50.593613 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:50.589509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:36209 (local address 127.4.49.3:51251)
0905 07:54:50.590265 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:50.590311 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:50.591038 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:50.591526 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:50.591535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:50.591629 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:50.592245 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0905 07:54:50.592247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:50.592283 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0905 07:54:50.592345 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:50.592584 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:50.592589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:50.592593 (+ 4us) client_negotiation.cc:769] Sending connection context
0905 07:54:50.592737 (+ 144us) client_negotiation.cc:240] Negotiation successful
0905 07:54:50.592928 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20250905 07:54:50.594348 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15195
I20250905 07:54:50.594453 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:50.594724 4292 external_mini_cluster.cc:1468] Setting key 36d478d0907580d80d827f34f1a53330
I20250905 07:54:50.595011 15324 heartbeater.cc:344] Connected to a master server at 127.4.49.62:36209
I20250905 07:54:50.595293 15324 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:50.595836 15324 heartbeater.cc:507] Master 127.4.49.62:36209 requested a full tablet report, sending...
I20250905 07:54:50.596860 14870 ts_manager.cc:194] Registered new tserver with Master: df00a0ee826d46e98b15c515d182a108 (127.4.49.3:41885)
I20250905 07:54:50.608697 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250905 07:54:50.611342 14910 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:54:50.610536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.4.49.62:36209)
0905 07:54:50.610655 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:50.610658 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:50.610844 (+ 186us) server_negotiation.cc:408] Connection header received
0905 07:54:50.610914 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:50.610917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:50.610960 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:50.611041 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:50.611202 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53406: BlockingRecv error: recv got EOF from 127.0.0.1:53406 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20250905 07:54:50.611678 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14931
I20250905 07:54:50.617019 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15063
I20250905 07:54:50.622335 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15195
I20250905 07:54:50.627478 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 14839
2025-09-05T07:54:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (814 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:50 dist-test-slave-mnfb krb5kdc[15335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:50 dist-test-slave-mnfb krb5kdc[15335](info): set up 2 sockets
Sep 05 07:54:50 dist-test-slave-mnfb krb5kdc[15335](info): commencing operation
krb5kdc: starting...
W20250905 07:54:52.459264 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:52 dist-test-slave-mnfb krb5kdc[15335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:52.592926 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39969
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:39969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:52.704888 15352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:52.705171 15352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:52.705255 15352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:52.708873 15352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:54:52.708961 15352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:52.708994 15352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:52.709019 15352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:54:52.709044 15352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:54:52.713837 15352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:39969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:39969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:52.715052 15352 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:52.716001 15352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:52.721966 15357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:52.721966 15358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:52.722020 15352 server_base.cc:1047] running on GCE node
W20250905 07:54:52.721967 15360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:52.722743 15352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:52.723838 15352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:52.725082 15352 hybrid_clock.cc:648] HybridClock initialized: now 1757058892725050 us; error 47 us; skew 500 ppm
Sep 05 07:54:52 dist-test-slave-mnfb krb5kdc[15335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:52.727953 15352 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:54:52.729297 15352 webserver.cc:480] Webserver started at http://127.4.49.62:40173/ using document root <none> and password file <none>
I20250905 07:54:52.729911 15352 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:52.729974 15352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:52.730175 15352 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:52.732111 15352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "ce1e075a79004512b5a92f8aa0b7958b"
format_stamp: "Formatted at 2025-09-05 07:54:52 on dist-test-slave-mnfb"
server_key: "f1be0ef953f42cf0065d001ec5cae4a1"
server_key_iv: "da24f59e136865cdca993cf3432d0a08"
server_key_version: "encryptionkey@0"
I20250905 07:54:52.732666 15352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "ce1e075a79004512b5a92f8aa0b7958b"
format_stamp: "Formatted at 2025-09-05 07:54:52 on dist-test-slave-mnfb"
server_key: "f1be0ef953f42cf0065d001ec5cae4a1"
server_key_iv: "da24f59e136865cdca993cf3432d0a08"
server_key_version: "encryptionkey@0"
I20250905 07:54:52.736424 15352 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:54:52.739084 15367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:52.740335 15352 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:52.740474 15352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "ce1e075a79004512b5a92f8aa0b7958b"
format_stamp: "Formatted at 2025-09-05 07:54:52 on dist-test-slave-mnfb"
server_key: "f1be0ef953f42cf0065d001ec5cae4a1"
server_key_iv: "da24f59e136865cdca993cf3432d0a08"
server_key_version: "encryptionkey@0"
I20250905 07:54:52.740585 15352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:52.752614 15352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:52.760638 15352 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:52.760862 15352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:52.768867 15352 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:39969
I20250905 07:54:52.768869 15419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:39969 every 8 connection(s)
I20250905 07:54:52.770076 15352 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:54:52.773411 15420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:52.779255 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15352
I20250905 07:54:52.779381 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:54:52.779616 4292 external_mini_cluster.cc:1468] Setting key db9424d379de06da2c772a34efe0ce8b
I20250905 07:54:52.779774 15420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: Bootstrap starting.
I20250905 07:54:52.782431 15420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:52.783244 15420 log.cc:826] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:52.785676 15420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: No bootstrap required, opened a new log
Sep 05 07:54:52 dist-test-slave-mnfb krb5kdc[15335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:52.788965 15420 raft_consensus.cc:357] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } }
I20250905 07:54:52.789213 15420 raft_consensus.cc:383] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:52.789271 15420 raft_consensus.cc:738] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1e075a79004512b5a92f8aa0b7958b, State: Initialized, Role: FOLLOWER
I20250905 07:54:52.789836 15420 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } }
I20250905 07:54:52.789980 15420 raft_consensus.cc:397] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:54:52.790033 15420 raft_consensus.cc:491] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:54:52.790163 15420 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:52.791347 15420 raft_consensus.cc:513] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } }
I20250905 07:54:52.791765 15420 leader_election.cc:304] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce1e075a79004512b5a92f8aa0b7958b; no voters:
I20250905 07:54:52.792148 15420 leader_election.cc:290] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:54:52.792270 15425 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:52.792587 15425 raft_consensus.cc:695] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [term 1 LEADER]: Becoming Leader. State: Replica: ce1e075a79004512b5a92f8aa0b7958b, State: Running, Role: LEADER
I20250905 07:54:52.792972 15425 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } }
I20250905 07:54:52.793516 15420 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:54:52.794648 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:52.781021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.4.49.62:39969)
0905 07:54:52.781531 (+ 510us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:52.781542 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:52.781584 (+ 42us) server_negotiation.cc:408] Connection header received
0905 07:54:52.782204 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:52.782224 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:52.782634 (+ 410us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:52.783032 (+ 398us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:52.784032 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:52.785162 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:52.785898 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:52.786196 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:52.788650 (+ 2454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:52.788680 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:52.788697 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:52.788773 (+ 76us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:52.791333 (+ 2560us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:52.791882 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:52.791890 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:52.791896 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:52.792033 (+ 137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:52.792379 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:52.792382 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:52.792384 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:52.792881 (+ 497us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:52.793105 (+ 224us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:52.793440 (+ 335us) server_negotiation.cc:300] Negotiation successful
0905 07:54:52.793782 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":169,"threads_started":1}
I20250905 07:54:52.794842 15427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1e075a79004512b5a92f8aa0b7958b. Latest consensus state: current_term: 1 leader_uuid: "ce1e075a79004512b5a92f8aa0b7958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } } }
I20250905 07:54:52.794942 15426 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce1e075a79004512b5a92f8aa0b7958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1e075a79004512b5a92f8aa0b7958b" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 39969 } } }
I20250905 07:54:52.794991 15427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:52.794999 15426 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b [sys.catalog]: This master's current role is: LEADER
I20250905 07:54:52.795389 15434 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:54:52.798931 15434 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:54:52.805634 15434 catalog_manager.cc:1349] Generated new cluster ID: 6de340896f4d4f9a87c824d4c41e61ba
I20250905 07:54:52.805740 15434 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:54:52.834668 15434 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:54:52.835670 15434 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:54:52.842476 15434 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: Generated new TSK 0
I20250905 07:54:52.843262 15434 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:52.894299 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39969
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:53.047789 15448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:53.048048 15448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:53.048138 15448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:53.051506 15448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:53.051579 15448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:53.051666 15448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:54:53.056267 15448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:53.057442 15448 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:53.058310 15448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:53.065518 15454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:53.065506 15456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:53.065622 15448 server_base.cc:1047] running on GCE node
W20250905 07:54:53.065506 15453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:53.066210 15448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:53.066817 15448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:53.067996 15448 hybrid_clock.cc:648] HybridClock initialized: now 1757058893067973 us; error 39 us; skew 500 ppm
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:53.070816 15448 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:54:53.072019 15448 webserver.cc:480] Webserver started at http://127.4.49.1:35119/ using document root <none> and password file <none>
I20250905 07:54:53.072647 15448 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:53.072705 15448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:53.072901 15448 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:53.074851 15448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "7b56084593304ad7af016b4b71684585"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "1af8a9dc5923cd9ee05446ca9eb3406f"
server_key_iv: "ed2411c245639ac1a9c584955c252dfd"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.075353 15448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "7b56084593304ad7af016b4b71684585"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "1af8a9dc5923cd9ee05446ca9eb3406f"
server_key_iv: "ed2411c245639ac1a9c584955c252dfd"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.079226 15448 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 07:54:53.081941 15463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.083151 15448 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 07:54:53.083307 15448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "7b56084593304ad7af016b4b71684585"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "1af8a9dc5923cd9ee05446ca9eb3406f"
server_key_iv: "ed2411c245639ac1a9c584955c252dfd"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.083448 15448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:53.100879 15448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:53.104200 15448 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:53.104404 15448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:53.105005 15448 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:53.105996 15448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:53.106052 15448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.106110 15448 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:53.106138 15448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.115378 15448 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:46425
I20250905 07:54:53.115406 15580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:46425 every 8 connection(s)
I20250905 07:54:53.116465 15448 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:54:53.122272 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15448
I20250905 07:54:53.122414 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:54:53.122678 4292 external_mini_cluster.cc:1468] Setting key 30d283f67309e7b4ca7e6ce0b4996a45
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:53.130275 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.118343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:39483 (local address 127.4.49.62:39969)
0905 07:54:53.118525 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.118530 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.119278 (+ 748us) server_negotiation.cc:408] Connection header received
0905 07:54:53.120169 (+ 891us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.120172 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.120227 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.120302 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:53.122000 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.122761 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.123394 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.123553 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.126404 (+ 2851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:53.126429 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:53.126432 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:53.126474 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:53.128184 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.128713 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.128717 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.128719 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.128784 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.129044 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.129047 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.129049 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.129275 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:53.129382 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.129933 (+ 551us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.130089 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250905 07:54:53.130888 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.118657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39969 (local address 127.4.49.1:39483)
0905 07:54:53.119127 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.119167 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.119943 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.120463 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.120472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.120906 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:53.121859 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.121875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.122884 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.122887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.123289 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.123297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.123511 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.124312 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:53.124352 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:53.126221 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:53.128315 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.128321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.128333 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.128601 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.128871 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.128873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.128875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.128964 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.129385 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:53.129391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:53.129655 (+ 264us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.129867 (+ 212us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.130116 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
I20250905 07:54:53.132112 15581 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39969
I20250905 07:54:53.132366 15581 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:53.132939 15581 heartbeater.cc:507] Master 127.4.49.62:39969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:54:53.134567 15384 ts_manager.cc:194] Registered new tserver with Master: 7b56084593304ad7af016b4b71684585 (127.4.49.1:46425)
I20250905 07:54:53.135821 15384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:39483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:53.168787 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39969
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:53.278445 15588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:53.278694 15588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:53.278761 15588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:53.282096 15588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:53.282181 15588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:53.282280 15588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:54:53.286808 15588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:53.287971 15588 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:53.288902 15588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:53.295392 15594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:53.295401 15596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:53.295509 15593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:53.295689 15588 server_base.cc:1047] running on GCE node
I20250905 07:54:53.296188 15588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:53.296789 15588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:53.297946 15588 hybrid_clock.cc:648] HybridClock initialized: now 1757058893297924 us; error 35 us; skew 500 ppm
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:53.300791 15588 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:54:53.301946 15588 webserver.cc:480] Webserver started at http://127.4.49.2:38795/ using document root <none> and password file <none>
I20250905 07:54:53.302526 15588 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:53.302577 15588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:53.302778 15588 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:53.304783 15588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "4b5f3a68351844ec9071156c7ddf9143"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "19c8ae49e2c28372347b06f59cba0cff"
server_key_iv: "53e7e4ace23f2c24e35096f8c14a5715"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.305276 15588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "4b5f3a68351844ec9071156c7ddf9143"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "19c8ae49e2c28372347b06f59cba0cff"
server_key_iv: "53e7e4ace23f2c24e35096f8c14a5715"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.309055 15588 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 07:54:53.311671 15604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.312825 15588 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:53.312954 15588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "4b5f3a68351844ec9071156c7ddf9143"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "19c8ae49e2c28372347b06f59cba0cff"
server_key_iv: "53e7e4ace23f2c24e35096f8c14a5715"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.313072 15588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:53.346849 15588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:53.350087 15588 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:53.350281 15588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:53.350893 15588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:53.351814 15588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:53.351862 15588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.351917 15588 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:53.351945 15588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.360566 15588 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:35321
I20250905 07:54:53.360613 15718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:35321 every 8 connection(s)
I20250905 07:54:53.361564 15588 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:54:53.365037 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15588
I20250905 07:54:53.365144 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:54:53.365397 4292 external_mini_cluster.cc:1468] Setting key 33e28463c8e8a9581e512cdfb69026d5
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
I20250905 07:54:53.374776 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.363332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:54677 (local address 127.4.49.62:39969)
0905 07:54:53.363473 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.363476 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.364213 (+ 737us) server_negotiation.cc:408] Connection header received
0905 07:54:53.365103 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.365107 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.365167 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.365249 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:53.367340 (+ 2091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.367868 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.368680 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.368895 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.370992 (+ 2097us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:53.371019 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:53.371023 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:53.371058 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:53.372718 (+ 1660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.373227 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.373231 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.373233 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.373291 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.373572 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.373576 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.373578 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.373811 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:53.373903 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.374456 (+ 553us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.374600 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:54:53.375423 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.363615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39969 (local address 127.4.49.2:54677)
0905 07:54:53.364088 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.364127 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.364899 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.365466 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.365480 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.365980 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:53.367180 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.367200 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.367992 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.367996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.368573 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.368582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.368748 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.369316 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:53.369342 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:53.370823 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:53.372852 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.372858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.372870 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.373120 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.373374 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.373376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.373378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.373465 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.373907 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:53.373912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:53.374182 (+ 270us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.374379 (+ 197us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.374694 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
I20250905 07:54:53.376574 15719 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39969
I20250905 07:54:53.376838 15719 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:53.377445 15719 heartbeater.cc:507] Master 127.4.49.62:39969 requested a full tablet report, sending...
I20250905 07:54:53.378633 15384 ts_manager.cc:194] Registered new tserver with Master: 4b5f3a68351844ec9071156c7ddf9143 (127.4.49.2:35321)
I20250905 07:54:53.379225 15384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:54677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:54:53.411208 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:39969
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:54:53.519523 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:54:53.519788 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:54:53.519874 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:54:53.523411 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:54:53.523510 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:54:53.523620 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
I20250905 07:54:53.528337 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:39969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:54:53.529529 15726 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:54:53.530416 15726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:54:53.537282 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:53.537268 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:54:53.537268 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:54:53.537638 15726 server_base.cc:1047] running on GCE node
I20250905 07:54:53.538013 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:54:53.538606 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:54:53.539767 15726 hybrid_clock.cc:648] HybridClock initialized: now 1757058893539751 us; error 36 us; skew 500 ppm
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:54:53.542551 15726 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:54:53.543720 15726 webserver.cc:480] Webserver started at http://127.4.49.3:38769/ using document root <none> and password file <none>
I20250905 07:54:53.544348 15726 fs_manager.cc:362] Metadata directory not provided
I20250905 07:54:53.544407 15726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:54:53.544610 15726 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:54:53.546468 15726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "daa49f6686504e3c913923c35974a7b6"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "368ac1138eb524f6cd264bb7c39db0b7"
server_key_iv: "0af982389f812cd89e82ce15bee8f7be"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.546984 15726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "daa49f6686504e3c913923c35974a7b6"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "368ac1138eb524f6cd264bb7c39db0b7"
server_key_iv: "0af982389f812cd89e82ce15bee8f7be"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.550611 15726 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250905 07:54:53.553025 15741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.554212 15726 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 07:54:53.554349 15726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "daa49f6686504e3c913923c35974a7b6"
format_stamp: "Formatted at 2025-09-05 07:54:53 on dist-test-slave-mnfb"
server_key: "368ac1138eb524f6cd264bb7c39db0b7"
server_key_iv: "0af982389f812cd89e82ce15bee8f7be"
server_key_version: "encryptionkey@0"
I20250905 07:54:53.554494 15726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:54:53.576982 15726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:54:53.580344 15726 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:54:53.580588 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:54:53.581171 15726 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:54:53.582094 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:54:53.582146 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.582199 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:54:53.582228 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:54:53.591284 15726 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:35251
I20250905 07:54:53.591354 15857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:35251 every 8 connection(s)
I20250905 07:54:53.592382 15726 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
I20250905 07:54:53.597280 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 15726
I20250905 07:54:53.597437 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:54:53.597739 4292 external_mini_cluster.cc:1468] Setting key 1ca0eb39a49f0edce70c619de9b79a9d
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058893, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:54:53.605103 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.594244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:50119 (local address 127.4.49.62:39969)
0905 07:54:53.594420 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.594424 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.595120 (+ 696us) server_negotiation.cc:408] Connection header received
0905 07:54:53.596002 (+ 882us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.596006 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.596112 (+ 106us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.596180 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:53.598138 (+ 1958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.598723 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.599386 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.599597 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.601721 (+ 2124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:53.601738 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:53.601740 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:53.601771 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:53.603137 (+ 1366us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.603627 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.603630 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.603632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.603676 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.603984 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.603988 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.603989 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.604197 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:53.604291 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.604851 (+ 560us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.604970 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250905 07:54:53.605798 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.594531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:39969 (local address 127.4.49.3:50119)
0905 07:54:53.594993 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.595033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.595791 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.596354 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.596362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.596780 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:54:53.597983 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.598001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.598841 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.598845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.599282 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.599291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.599457 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.600019 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:54:53.600044 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:54:53.601552 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:54:53.603265 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.603270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.603280 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.603540 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.603768 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:54:53.603770 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:54:53.603772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:54:53.603870 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:54:53.604326 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:54:53.604332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:54:53.604605 (+ 273us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.604798 (+ 193us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.605034 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20250905 07:54:53.607012 15858 heartbeater.cc:344] Connected to a master server at 127.4.49.62:39969
I20250905 07:54:53.607245 15858 heartbeater.cc:461] Registering TS with master...
I20250905 07:54:53.607780 15858 heartbeater.cc:507] Master 127.4.49.62:39969 requested a full tablet report, sending...
I20250905 07:54:53.608912 15384 ts_manager.cc:194] Registered new tserver with Master: daa49f6686504e3c913923c35974a7b6 (127.4.49.3:35251)
I20250905 07:54:53.609510 15384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:50119
I20250905 07:54:53.611893 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:54:53.620677 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.613874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.4.49.62:39969)
0905 07:54:53.613999 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.614003 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.614203 (+ 200us) server_negotiation.cc:408] Connection header received
0905 07:54:53.614344 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.614346 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.614395 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.614477 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:54:53.615352 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.615883 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.616643 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.616786 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.617924 (+ 1138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:54:53.617942 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:54:53.617945 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:54:53.617970 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:54:53.619311 (+ 1341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.619696 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.619699 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.619700 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.619742 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:54:53.619906 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:54:53.619908 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:54:53.619909 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:54:53.620130 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:54:53.620230 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.620407 (+ 177us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.620517 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:54:53.623390 15384 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:54:53.625888 15384 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:54:53.638593 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.634231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:46425 (local address 127.0.0.1:33562)
0905 07:54:53.634819 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.634860 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.635013 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.635757 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.635763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.635786 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:53.636113 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.636123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.637428 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.637432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.638086 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.638095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.638196 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.638254 (+ 58us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.638334 (+ 80us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.638420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":101,"threads_started":1}
I20250905 07:54:53.639559 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.635008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.4.49.3:35251)
0905 07:54:53.635334 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.635340 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.635477 (+ 137us) server_negotiation.cc:408] Connection header received
0905 07:54:53.635612 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.635617 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.635776 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.636413 (+ 637us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:53.636578 (+ 165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.637324 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.638690 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.639139 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.639267 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.639330 (+ 63us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.639419 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
I20250905 07:54:53.639881 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.634356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.4.49.1:46425)
0905 07:54:53.634657 (+ 301us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.634664 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.635142 (+ 478us) server_negotiation.cc:408] Connection header received
0905 07:54:53.635242 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.635248 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.635437 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.635572 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:53.636238 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.637296 (+ 1058us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.638976 (+ 1680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.639501 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.639590 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.639652 (+ 62us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.639757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20250905 07:54:53.638762 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.634272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:35251 (local address 127.0.0.1:41756)
0905 07:54:53.635375 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.635388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.635498 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.636036 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.636039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.636091 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:53.636296 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.636302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.637477 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.637481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.638370 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.638379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.638469 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.638488 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.638534 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.638591 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":83,"threads_started":1}
I20250905 07:54:53.640178 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.635365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.4.49.2:35321)
0905 07:54:53.635741 (+ 376us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.635747 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.636071 (+ 324us) server_negotiation.cc:408] Connection header received
0905 07:54:53.636149 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.636153 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.636313 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.636458 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:53.636829 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.637964 (+ 1135us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.639236 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.639816 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.639889 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.639940 (+ 51us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.640022 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":105,"threads_started":1}
I20250905 07:54:53.639432 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.634951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:35321 (local address 127.0.0.1:33488)
0905 07:54:53.635876 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.635887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.635963 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.636450 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.636454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.636471 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:53.636697 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.636705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.638083 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.638087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.639086 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.639095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.639200 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.639216 (+ 16us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.639261 (+ 45us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.639314 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":88,"threads_started":1}
I20250905 07:54:53.642999 15514 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea775711314cdeb470bf384904058c (DEFAULT_TABLE table=test-table [id=0442be02731a4f1987af767224e12c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:53.643901 15514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea775711314cdeb470bf384904058c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:53.646564 15792 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea775711314cdeb470bf384904058c (DEFAULT_TABLE table=test-table [id=0442be02731a4f1987af767224e12c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:53.647045 15653 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea775711314cdeb470bf384904058c (DEFAULT_TABLE table=test-table [id=0442be02731a4f1987af767224e12c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:54:53.647505 15792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea775711314cdeb470bf384904058c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:53.647962 15653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea775711314cdeb470bf384904058c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:54:53.650710 15876 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: Bootstrap starting.
I20250905 07:54:53.653136 15877 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Bootstrap starting.
I20250905 07:54:53.653762 15876 tablet_bootstrap.cc:654] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:53.654251 15878 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Bootstrap starting.
I20250905 07:54:53.654647 15876 log.cc:826] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:53.655342 15877 tablet_bootstrap.cc:654] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:53.656083 15877 log.cc:826] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:53.656276 15878 tablet_bootstrap.cc:654] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Neither blocks nor log segments found. Creating new log.
I20250905 07:54:53.656805 15876 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: No bootstrap required, opened a new log
I20250905 07:54:53.657013 15876 ts_tablet_manager.cc:1397] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250905 07:54:53.657054 15878 log.cc:826] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Log is configured to *not* fsync() on all Append() calls
I20250905 07:54:53.658049 15877 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: No bootstrap required, opened a new log
I20250905 07:54:53.658313 15877 ts_tablet_manager.cc:1397] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250905 07:54:53.658843 15878 tablet_bootstrap.cc:492] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: No bootstrap required, opened a new log
I20250905 07:54:53.659029 15878 ts_tablet_manager.cc:1397] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250905 07:54:53.660251 15876 raft_consensus.cc:357] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.660498 15876 raft_consensus.cc:383] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:53.660593 15876 raft_consensus.cc:738] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b56084593304ad7af016b4b71684585, State: Initialized, Role: FOLLOWER
I20250905 07:54:53.661075 15876 consensus_queue.cc:260] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.661270 15877 raft_consensus.cc:357] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.661538 15877 raft_consensus.cc:383] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:53.661628 15877 raft_consensus.cc:738] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa49f6686504e3c913923c35974a7b6, State: Initialized, Role: FOLLOWER
I20250905 07:54:53.661832 15581 heartbeater.cc:499] Master 127.4.49.62:39969 was elected leader, sending a full tablet report...
I20250905 07:54:53.662043 15877 consensus_queue.cc:260] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.662452 15878 raft_consensus.cc:357] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.662616 15858 heartbeater.cc:499] Master 127.4.49.62:39969 was elected leader, sending a full tablet report...
I20250905 07:54:53.662640 15878 raft_consensus.cc:383] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:54:53.662688 15878 raft_consensus.cc:738] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5f3a68351844ec9071156c7ddf9143, State: Initialized, Role: FOLLOWER
I20250905 07:54:53.662896 15877 ts_tablet_manager.cc:1428] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 07:54:53.663065 15878 consensus_queue.cc:260] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.663631 15876 ts_tablet_manager.cc:1428] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250905 07:54:53.663868 15878 ts_tablet_manager.cc:1428] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 07:54:53.664214 15719 heartbeater.cc:499] Master 127.4.49.62:39969 was elected leader, sending a full tablet report...
I20250905 07:54:53.799194 15884 raft_consensus.cc:491] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:53.799410 15884 raft_consensus.cc:513] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.800621 15884 leader_election.cc:290] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b56084593304ad7af016b4b71684585 (127.4.49.1:46425), daa49f6686504e3c913923c35974a7b6 (127.4.49.3:35251)
I20250905 07:54:53.803534 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.800900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:35251 (local address 127.4.49.2:34335)
0905 07:54:53.801123 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.801139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.801254 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.801577 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.801581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.801601 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:53.801819 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.801826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.802532 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.802536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.803125 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.803133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.803245 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.803270 (+ 25us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.803318 (+ 48us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.803381 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20250905 07:54:53.803753 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.800900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:46425 (local address 127.4.49.2:47179)
0905 07:54:53.801437 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.801448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0905 07:54:53.801532 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:54:53.801764 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:54:53.801767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:54:53.801785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:54:53.802021 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.802027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.802781 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.802784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:54:53.803358 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:54:53.803366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.803474 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.803493 (+ 19us) client_negotiation.cc:769] Sending connection context
0905 07:54:53.803540 (+ 47us) client_negotiation.cc:240] Negotiation successful
0905 07:54:53.803599 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":172,"thread_start_us":151,"threads_started":1}
I20250905 07:54:53.803988 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.801020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:34335 (local address 127.4.49.3:35251)
0905 07:54:53.801137 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.801142 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.801218 (+ 76us) server_negotiation.cc:408] Connection header received
0905 07:54:53.801387 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.801392 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.801443 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.801546 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:53.801931 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.802415 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.803240 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.803725 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.803770 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.803820 (+ 50us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.803875 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250905 07:54:53.804239 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.801011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:47179 (local address 127.4.49.1:46425)
0905 07:54:53.801145 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.801149 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.801534 (+ 385us) server_negotiation.cc:408] Connection header received
0905 07:54:53.801617 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.801620 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.801668 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.801730 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:54:53.802142 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.802648 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.803485 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.803953 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.803993 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.804043 (+ 50us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.804127 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250905 07:54:53.804630 15812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea775711314cdeb470bf384904058c" candidate_uuid: "4b5f3a68351844ec9071156c7ddf9143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa49f6686504e3c913923c35974a7b6" is_pre_election: true
I20250905 07:54:53.804733 15534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea775711314cdeb470bf384904058c" candidate_uuid: "4b5f3a68351844ec9071156c7ddf9143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b56084593304ad7af016b4b71684585" is_pre_election: true
I20250905 07:54:53.804960 15812 raft_consensus.cc:2466] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5f3a68351844ec9071156c7ddf9143 in term 0.
I20250905 07:54:53.804965 15534 raft_consensus.cc:2466] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5f3a68351844ec9071156c7ddf9143 in term 0.
I20250905 07:54:53.805475 15605 leader_election.cc:304] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5f3a68351844ec9071156c7ddf9143, 7b56084593304ad7af016b4b71684585; no voters:
I20250905 07:54:53.805720 15884 raft_consensus.cc:2802] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:54:53.805799 15884 raft_consensus.cc:491] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:54:53.805845 15884 raft_consensus.cc:3058] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:53.806783 15884 raft_consensus.cc:513] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.807138 15884 leader_election.cc:290] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [CANDIDATE]: Term 1 election: Requested vote from peers 7b56084593304ad7af016b4b71684585 (127.4.49.1:46425), daa49f6686504e3c913923c35974a7b6 (127.4.49.3:35251)
I20250905 07:54:53.807487 15534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea775711314cdeb470bf384904058c" candidate_uuid: "4b5f3a68351844ec9071156c7ddf9143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b56084593304ad7af016b4b71684585"
I20250905 07:54:53.807515 15812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea775711314cdeb470bf384904058c" candidate_uuid: "4b5f3a68351844ec9071156c7ddf9143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa49f6686504e3c913923c35974a7b6"
I20250905 07:54:53.807598 15534 raft_consensus.cc:3058] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:53.807658 15812 raft_consensus.cc:3058] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:54:53.808719 15812 raft_consensus.cc:2466] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5f3a68351844ec9071156c7ddf9143 in term 1.
I20250905 07:54:53.809031 15534 raft_consensus.cc:2466] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5f3a68351844ec9071156c7ddf9143 in term 1.
I20250905 07:54:53.809068 15607 leader_election.cc:304] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5f3a68351844ec9071156c7ddf9143, daa49f6686504e3c913923c35974a7b6; no voters:
I20250905 07:54:53.809268 15884 raft_consensus.cc:2802] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:54:53.809506 15884 raft_consensus.cc:695] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 1 LEADER]: Becoming Leader. State: Replica: 4b5f3a68351844ec9071156c7ddf9143, State: Running, Role: LEADER
I20250905 07:54:53.809830 15884 consensus_queue.cc:237] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } }
I20250905 07:54:53.813181 15383 catalog_manager.cc:5582] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5f3a68351844ec9071156c7ddf9143 (127.4.49.2). New cstate: current_term: 1 leader_uuid: "4b5f3a68351844ec9071156c7ddf9143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5f3a68351844ec9071156c7ddf9143" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 35321 } health_report { overall_health: HEALTHY } } }
I20250905 07:54:53.831365 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.828418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.4.49.2:35321)
0905 07:54:53.828570 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0905 07:54:53.828574 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:54:53.828588 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:54:53.828668 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:54:53.828671 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:54:53.828729 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:54:53.828810 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:54:53.829136 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.829628 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:54:53.830262 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:54:53.830411 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:54:53.830523 (+ 112us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:54:53.830972 (+ 449us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:54:53.831065 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0905 07:54:53.831169 (+ 104us) server_negotiation.cc:300] Negotiation successful
0905 07:54:53.831221 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250905 07:54:53.836885 15812 raft_consensus.cc:1273] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 1 FOLLOWER]: Refusing update from remote peer 4b5f3a68351844ec9071156c7ddf9143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:53.836884 15534 raft_consensus.cc:1273] T a9ea775711314cdeb470bf384904058c P 7b56084593304ad7af016b4b71684585 [term 1 FOLLOWER]: Refusing update from remote peer 4b5f3a68351844ec9071156c7ddf9143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:54:53.837563 15886 consensus_queue.cc:1035] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b56084593304ad7af016b4b71684585" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:54:53.837735 15884 consensus_queue.cc:1035] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa49f6686504e3c913923c35974a7b6" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:54:53.843294 15859 tablet.cc:2378] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:54:53.844277 15889 mvcc.cc:204] Tried to move back new op lower bound from 7196913229148864512 to 7196913229047660544. Current Snapshot: MvccSnapshot[applied={T|T < 7196913229148864512}]
I20250905 07:54:53.852344 15384 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38860:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:53.852538 15384 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38860:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:54:53.855034 15384 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ce1e075a79004512b5a92f8aa0b7958b: Sending DeleteTablet for 3 replicas of tablet a9ea775711314cdeb470bf384904058c
I20250905 07:54:53.855896 15792 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea775711314cdeb470bf384904058c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:53 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:41756
I20250905 07:54:53.855894 15653 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea775711314cdeb470bf384904058c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:53 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:33488
I20250905 07:54:53.856463 15514 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea775711314cdeb470bf384904058c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:54:53 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:33562
I20250905 07:54:53.856644 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15448
I20250905 07:54:53.857234 15899 tablet_replica.cc:331] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: stopping tablet replica
I20250905 07:54:53.856688 15900 tablet_replica.cc:331] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: stopping tablet replica
I20250905 07:54:53.857550 15899 raft_consensus.cc:2241] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:54:53.857622 15900 raft_consensus.cc:2241] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:54:53.857911 15899 raft_consensus.cc:2270] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:53.857967 15900 raft_consensus.cc:2270] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:54:53.859526 15900 ts_tablet_manager.cc:1905] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:53.860024 15899 ts_tablet_manager.cc:1905] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:54:53.862334 15900 ts_tablet_manager.cc:1918] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:53.862468 15900 log.cc:1199] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/a9ea775711314cdeb470bf384904058c
I20250905 07:54:53.862637 15899 ts_tablet_manager.cc:1918] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:54:53.862759 15899 log.cc:1199] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1757058797595744-4292-0/minicluster-data/ts-1/wal/wals/a9ea775711314cdeb470bf384904058c
I20250905 07:54:53.862839 15900 ts_tablet_manager.cc:1939] T a9ea775711314cdeb470bf384904058c P daa49f6686504e3c913923c35974a7b6: Deleting consensus metadata
I20250905 07:54:53.863087 15899 ts_tablet_manager.cc:1939] T a9ea775711314cdeb470bf384904058c P 4b5f3a68351844ec9071156c7ddf9143: Deleting consensus metadata
I20250905 07:54:53.864202 15368 catalog_manager.cc:4928] TS 4b5f3a68351844ec9071156c7ddf9143 (127.4.49.2:35321): tablet a9ea775711314cdeb470bf384904058c (table test-table [id=0442be02731a4f1987af767224e12c2e]) successfully deleted
I20250905 07:54:53.864284 15370 catalog_manager.cc:4928] TS daa49f6686504e3c913923c35974a7b6 (127.4.49.3:35251): tablet a9ea775711314cdeb470bf384904058c (table test-table [id=0442be02731a4f1987af767224e12c2e]) successfully deleted
W20250905 07:54:53.864837 15368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:46425 (error 108)
I20250905 07:54:53.865195 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15588
I20250905 07:54:53.866428 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:54:53.866119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:46425 (local address 127.0.0.1:33566)
0905 07:54:53.866272 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0905 07:54:53.866339 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:46425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20250905 07:54:53.866783 15368 catalog_manager.cc:4671] TS 7b56084593304ad7af016b4b71684585 (127.4.49.1:46425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9ea775711314cdeb470bf384904058c: Network error: Client connection negotiation failed: client connection to 127.4.49.1:46425: connect: Connection refused (error 111)
I20250905 07:54:53.870963 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15726
I20250905 07:54:53.876322 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 15352
2025-09-05T07:54:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3249 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15904](info): set up 2 sockets
Sep 05 07:54:53 dist-test-slave-mnfb krb5kdc[15904](info): commencing operation
krb5kdc: starting...
W20250905 07:54:55.706998 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.809s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:54:55 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:54:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:54:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangeradmin_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangeradmin_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangerlookup_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangerlookup_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
I20250905 07:54:55.856729 4292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5YGlfE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/postgres -l logfile start
2025-09-05 07:54:58.131 UTC [15936] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-05 07:54:58.131 UTC [15936] LOG: listening on IPv4 address "127.4.49.20", port 51689
2025-09-05 07:54:58.134 UTC [15936] LOG: listening on Unix socket "/tmp/.s.PGSQL.51689"
2025-09-05 07:54:58.138 UTC [15941] LOG: database system was shut down at 2025-09-05 07:54:57 UTC
2025-09-05 07:54:58.142 UTC [15936] LOG: database system is ready to accept connections
I20250905 07:54:59.898608 4292 mini_postgres.cc:96] Postgres bound to 51689
2025-09-05 07:54:59.904 UTC [15947] FATAL: database "slave" does not exist
127.4.49.20:51689 - accepting connections
I20250905 07:54:59.904839 4292 mini_ranger.cc:161] Starting Ranger...
I20250905 07:54:59.924583 4292 mini_ranger.cc:84] Created miniranger Postgres user
I20250905 07:54:59.992914 4292 mini_ranger.cc:87] Created ranger Postgres database
I20250905 07:54:59.993027 4292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin
2025-09-05 07:55:00,679 [I] DB FLAVOR :POSTGRES
2025-09-05 07:55:00,679 [I] --------- Verifying Ranger DB connection ---------
2025-09-05 07:55:00,679 [I] Checking connection..
2025-09-05 07:55:00,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-05 07:55:01,041 [I] Checking connection passed.
2025-09-05 07:55:01,042 [I] --------- Verifying version history table ---------
2025-09-05 07:55:01,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:01,369 [I] Table x_db_version_h does not exist in database ranger
2025-09-05 07:55:01,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:01,699 [I] Table x_db_version_h does not exist in database ranger
2025-09-05 07:55:01,700 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-05 07:55:01,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-05 07:55:02.013 UTC [16015] WARNING: there is no transaction in progress
2025-09-05 07:55:02,019 [I] create_dbversion_catalog.sql file imported successfully
2025-09-05 07:55:02,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:02,365 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-05 07:55:02,366 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-05 07:55:02,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-05 07:55:02,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-05 07:55:03,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mnfb.c.gcp-upstream.internal','N') ;"
2025-09-05 07:55:03,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-05 07:55:03,692 [I] Table x_portal_user does not exist in database ranger
2025-09-05 07:55:03,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-05 07:55:04,023 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-05 07:55:04,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-05 07:55:04,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-05 07:55:04,686 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-05 07:55:04,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-05 07:55:05.621 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.636 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.652 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.668 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.684 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.702 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.768 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.775 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.786 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.800 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.811 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.821 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.829 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.834 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.839 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:05.844 UTC [16133] WARNING: there is no transaction in progress
2025-09-05 07:55:06,302 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-05 07:55:06,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-05 07:55:06,626 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-05 07:55:06,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-05 07:55:06,976 [I] Table x_portal_user already exists in database 'ranger'
2025-09-05 07:55:06,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-05 07:55:07,306 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-05 07:55:07,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-05 07:55:07,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-05 07:55:08,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mnfb.c.gcp-upstream.internal';"
2025-09-05 07:55:08,346 [I] CORE_DB_SCHEMA import status has been updated
2025-09-05 07:55:08,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-05 07:55:08,685 [I] DB_PATCHES have already been applied
I20250905 07:55:08.691800 4292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5YGlfE/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250905 07:55:08.691877 4292 mini_ranger.cc:193] Using host: 127.4.49.20
I20250905 07:55:08.694775 4292 mini_ranger.cc:227] Ranger admin URL: http://127.4.49.20:51355
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 05, 2025 7:55:09 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp].
Sep 05, 2025 7:55:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 05, 2025 7:55:09 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-51355"]
Sep 05, 2025 7:55:10 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 05, 2025 7:55:10 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 05, 2025 7:55:10 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 05, 2025 7:55:10 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 05, 2025 7:55:10 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20250905 07:55:10.742466 4292 mini_ranger.cc:160] Time spent starting Ranger: real 10.838s user 0.008s sys 0.003s
Sep 05, 2025 7:55:11 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 05, 2025 7:55:11 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 05, 2025 7:55:11 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Sep 05, 2025 7:55:11 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Sep 05, 2025 7:55:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 05, 2025 7:55:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.AssetREST
Sep 05, 2025 7:55:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 05, 2025 7:55:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 05, 2025 7:55:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 05, 2025 7:55:32 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-51355"]
I20250905 07:55:33.584406 4292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangerkms_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/rangerkms_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.49.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.49.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 05 07:55:33 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058933, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250905 07:55:33.661144 4292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250905 07:55:33.675881 4292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250905 07:55:33.898533 4292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250905 07:55:33.898638 4292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms
I20250905 07:55:33.898666 4292 mini_ranger_kms.cc:227] Using postgres at 127.4.49.20:51689
2025-09-05 07:55:35,176 [I] DB FLAVOR :POSTGRES
2025-09-05 07:55:35,176 [I] --------- Verifying Ranger DB connection ---------
2025-09-05 07:55:35,176 [I] Checking connection
2025-09-05 07:55:35,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-05 07:55:35,668 [I] connection success
2025-09-05 07:55:35,669 [I] --------- Verifying Ranger DB tables ---------
2025-09-05 07:55:35,669 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-05 07:55:35,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-05 07:55:35,997 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-05 07:55:35,997 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-05 07:55:35,997 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-05 07:55:35,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:51689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-05 07:55:36,336 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250905 07:55:36.813287 4292 mini_ranger_kms.cc:310] Created kms service
I20250905 07:55:37.003322 4292 mini_ranger_kms.cc:326] Created kudu user
I20250905 07:55:37.044512 4292 mini_ranger_kms.cc:343] Created rangerkms user
Sep 05 07:55:37 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058933, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:37.426712 4292 mini_ranger_kms.cc:384] Added ranger policy
I20250905 07:55:37.426872 4292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms:/tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250905 07:55:37.426893 4292 mini_ranger_kms.cc:240] Using host: 127.4.49.20
I20250905 07:55:37.430265 4292 mini_ranger_kms.cc:276] Ranger KMS PID: 16329
I20250905 07:55:37.430352 4292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.49.20:43783
Sep 05, 2025 7:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 05, 2025 7:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 05, 2025 7:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 05, 2025 7:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 05, 2025 7:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp].
Sep 05, 2025 7:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 05, 2025 7:55:39 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-43783"]
Sep 05, 2025 7:55:39 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 05, 2025 7:55:39 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
I20250905 07:55:39.422221 4292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.761s user 0.000s sys 0.995s
I20250905 07:55:39.422377 4292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250905 07:55:39.422468 4292 mini_ranger_kms.cc:401] 127.4.49.20:43783/kms/v1/keys
Sep 05, 2025 7:55:39 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 05, 2025 7:55:39 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 05 07:55:39 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1757058939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 05 07:55:40 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
Sep 05 07:55:40 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
Sep 05 07:55:43 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:55:43.646661 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44209
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:44209
--ranger_config_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:43783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:55:43.798290 16383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:55:43.798555 16383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:55:43.798617 16383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:55:43.802066 16383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:55:43.802155 16383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:55:43.802176 16383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:55:43.802194 16383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:55:43.802211 16383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:55:43.807051 16383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.49.20:43783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:44209
--ranger_config_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:55:43.808260 16383 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:55:43.809154 16383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:55:43.815225 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:43.815443 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:43.815232 16388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:55:43.816854 16383 server_base.cc:1047] running on GCE node
I20250905 07:55:43.817471 16383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:55:43.818548 16383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:55:43.819716 16383 hybrid_clock.cc:648] HybridClock initialized: now 1757058943819703 us; error 57 us; skew 500 ppm
Sep 05 07:55:43 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058943, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:55:43.822770 16383 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:55:43.824159 16383 webserver.cc:480] Webserver started at http://127.4.49.62:38403/ using document root <none> and password file <none>
I20250905 07:55:43.824757 16383 fs_manager.cc:362] Metadata directory not provided
I20250905 07:55:43.824831 16383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:55:43.825042 16383 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:55:43 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058943, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:43.884666 16383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "00d0519ff7ad4be78fecf7c607c028f6"
format_stamp: "Formatted at 2025-09-05 07:55:43 on dist-test-slave-mnfb"
server_key: "2c6dd2a372bc3dd9ea5a1088f765552a"
server_key_iv: "38488fe2fb832991efdf2db9c5489762"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:43.885442 16383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "00d0519ff7ad4be78fecf7c607c028f6"
format_stamp: "Formatted at 2025-09-05 07:55:43 on dist-test-slave-mnfb"
server_key: "2c6dd2a372bc3dd9ea5a1088f765552a"
server_key_iv: "38488fe2fb832991efdf2db9c5489762"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:43.890534 16383 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 07:55:43.905709 16399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:43.906896 16383 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 07:55:43.907037 16383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "00d0519ff7ad4be78fecf7c607c028f6"
format_stamp: "Formatted at 2025-09-05 07:55:43 on dist-test-slave-mnfb"
server_key: "2c6dd2a372bc3dd9ea5a1088f765552a"
server_key_iv: "38488fe2fb832991efdf2db9c5489762"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:43.907145 16383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:55:43.923686 16383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:55:43.927119 16383 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:55:43.927325 16383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:55:43.942759 16383 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:44209
I20250905 07:55:43.942836 16452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:44209 every 8 connection(s)
I20250905 07:55:43.944152 16383 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:55:43.945377 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 16383
I20250905 07:55:43.945516 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:55:43.945746 4292 external_mini_cluster.cc:1468] Setting key 0647f889589617f3c0703aa2dd4f7f00
I20250905 07:55:43.949394 16453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 05 07:55:43 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:55:43.958456 16453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: Bootstrap starting.
I20250905 07:55:43.960258 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:43.947122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.4.49.62:44209)
0905 07:55:43.947731 (+ 609us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:43.947742 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:43.948384 (+ 642us) server_negotiation.cc:408] Connection header received
0905 07:55:43.949508 (+ 1124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:43.949540 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:43.949886 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:43.950224 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:55:43.951302 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:43.952328 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:43.953014 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:43.953334 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:43.955618 (+ 2284us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:55:43.955669 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:55:43.955682 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:55:43.955728 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:55:43.957622 (+ 1894us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:43.958065 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:43.958071 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:43.958076 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:43.958154 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:43.958373 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:43.958377 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:43.958379 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:43.958794 (+ 415us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:55:43.958979 (+ 185us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:43.959276 (+ 297us) server_negotiation.cc:300] Negotiation successful
0905 07:55:43.959533 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":217,"threads_started":1}
I20250905 07:55:43.963299 16453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: Neither blocks nor log segments found. Creating new log.
I20250905 07:55:43.965058 16453 log.cc:826] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: Log is configured to *not* fsync() on all Append() calls
I20250905 07:55:43.967604 16453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: No bootstrap required, opened a new log
I20250905 07:55:43.972591 16453 raft_consensus.cc:357] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } }
I20250905 07:55:43.972877 16453 raft_consensus.cc:383] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:55:43.972936 16453 raft_consensus.cc:738] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d0519ff7ad4be78fecf7c607c028f6, State: Initialized, Role: FOLLOWER
I20250905 07:55:43.973450 16453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } }
I20250905 07:55:43.973644 16453 raft_consensus.cc:397] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:55:43.973713 16453 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:55:43.973870 16453 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:55:43.975334 16453 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } }
I20250905 07:55:43.975839 16453 leader_election.cc:304] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d0519ff7ad4be78fecf7c607c028f6; no voters:
I20250905 07:55:43.976222 16453 leader_election.cc:290] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:55:43.976449 16458 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:55:43.976743 16458 raft_consensus.cc:695] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [term 1 LEADER]: Becoming Leader. State: Replica: 00d0519ff7ad4be78fecf7c607c028f6, State: Running, Role: LEADER
I20250905 07:55:43.977129 16458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } }
I20250905 07:55:43.977615 16453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:55:43.978781 16460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d0519ff7ad4be78fecf7c607c028f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } } }
I20250905 07:55:43.978901 16460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [sys.catalog]: This master's current role is: LEADER
I20250905 07:55:43.979456 16458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d0519ff7ad4be78fecf7c607c028f6. Latest consensus state: current_term: 1 leader_uuid: "00d0519ff7ad4be78fecf7c607c028f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0519ff7ad4be78fecf7c607c028f6" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 44209 } } }
I20250905 07:55:43.979704 16458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6 [sys.catalog]: This master's current role is: LEADER
I20250905 07:55:43.980607 16465 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:55:43.984472 16465 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:55:43.985047 16453 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250905 07:55:43.997675 16465 catalog_manager.cc:1349] Generated new cluster ID: fa65b8a064bc4131b2900f6a0529b4ab
I20250905 07:55:43.997838 16465 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:55:44.020977 16465 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:55:44.022445 16465 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:55:44.040843 16465 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: Generated new TSK 0
I20250905 07:55:44.041692 16465 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:55:45.072376 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:43783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44209
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:55:45.247220 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:55:45.247462 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:55:45.247536 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:55:45.252266 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:55:45.252342 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:55:45.252426 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:55:45.257197 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.49.20:43783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:55:45.258364 16499 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:55:45.259234 16499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:55:45.266080 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:45.266201 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:45.266113 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:55:45.266508 16499 server_base.cc:1047] running on GCE node
I20250905 07:55:45.266899 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:55:45.267470 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:55:45.268661 16499 hybrid_clock.cc:648] HybridClock initialized: now 1757058945268625 us; error 50 us; skew 500 ppm
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:55:45.271780 16499 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:55:45.272987 16499 webserver.cc:480] Webserver started at http://127.4.49.1:37533/ using document root <none> and password file <none>
I20250905 07:55:45.273540 16499 fs_manager.cc:362] Metadata directory not provided
I20250905 07:55:45.273595 16499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:55:45.273788 16499 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:45.291164 16499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "06bd3cbe15ce48159c3110bab7448117"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "8988c3991bce00033f2d63ddf7d1f1e2"
server_key_iv: "c4f14ab0c7c88508d223c00cc1b37645"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.291823 16499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "06bd3cbe15ce48159c3110bab7448117"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "8988c3991bce00033f2d63ddf7d1f1e2"
server_key_iv: "c4f14ab0c7c88508d223c00cc1b37645"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.298960 16499 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20250905 07:55:45.316902 16516 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:45.318836 16499 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 07:55:45.319006 16499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "06bd3cbe15ce48159c3110bab7448117"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "8988c3991bce00033f2d63ddf7d1f1e2"
server_key_iv: "c4f14ab0c7c88508d223c00cc1b37645"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.319159 16499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:55:45.365116 16499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:55:45.372162 16499 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:55:45.372430 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:55:45.373342 16499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:55:45.374586 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:55:45.374706 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:45.374847 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:55:45.374923 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:45.395262 16499 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:46133
I20250905 07:55:45.395627 16631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:46133 every 8 connection(s)
I20250905 07:55:45.396824 16499 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:55:45.404671 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 16499
I20250905 07:55:45.404824 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:55:45.405097 4292 external_mini_cluster.cc:1468] Setting key a3a2e9b331e42a29150749f7ddfbdbc8
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
I20250905 07:55:45.427315 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:45.404242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:48643 (local address 127.4.49.62:44209)
0905 07:55:45.410276 (+ 6034us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:45.410282 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:45.410308 (+ 26us) server_negotiation.cc:408] Connection header received
0905 07:55:45.410363 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:45.410367 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:45.410429 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:45.412150 (+ 1721us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:55:45.412729 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.413634 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:45.415399 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.418298 (+ 2899us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:45.418450 (+ 152us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:55:45.418488 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:55:45.418492 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:55:45.418516 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:55:45.421061 (+ 2545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:45.424385 (+ 3324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:45.424391 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:45.424393 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:45.424483 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:45.425003 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:45.425006 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:45.425009 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:45.425223 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:55:45.426833 (+ 1610us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:45.426956 (+ 123us) server_negotiation.cc:300] Negotiation successful
0905 07:55:45.427125 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5912,"thread_start_us":100,"threads_started":1}
I20250905 07:55:45.427441 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:45.403115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44209 (local address 127.4.49.1:48643)
0905 07:55:45.403832 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:45.403884 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:45.405409 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:45.410893 (+ 5484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:45.410908 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:45.411348 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:55:45.412486 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:45.412510 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.413812 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:45.413816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:45.414610 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:45.414623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.414940 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:45.415757 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:55:45.415894 (+ 137us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:55:45.418023 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:55:45.421264 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:45.421273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:45.421290 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:45.421859 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:45.424639 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:45.424644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:45.424646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:45.424862 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:45.425369 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:55:45.425378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:55:45.425753 (+ 375us) client_negotiation.cc:769] Sending connection context
0905 07:55:45.425981 (+ 228us) client_negotiation.cc:240] Negotiation successful
0905 07:55:45.426362 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":201,"threads_started":1}
I20250905 07:55:45.429039 16632 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44209
I20250905 07:55:45.429359 16632 heartbeater.cc:461] Registering TS with master...
I20250905 07:55:45.430102 16632 heartbeater.cc:507] Master 127.4.49.62:44209 requested a full tablet report, sending...
I20250905 07:55:45.432192 16416 ts_manager.cc:194] Registered new tserver with Master: 06bd3cbe15ce48159c3110bab7448117 (127.4.49.1:46133)
I20250905 07:55:45.433527 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:48643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:55:45.465147 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:43783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44209
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20250905 07:55:45.647531 16642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:55:45.647887 16642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:55:45.647981 16642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:55:45.652766 16642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:55:45.652870 16642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:55:45.652995 16642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:55:45.659373 16642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.49.20:43783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:55:45.661042 16642 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:55:45.662194 16642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:55:45.670900 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:45.670910 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:45.671094 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:55:45.671617 16642 server_base.cc:1047] running on GCE node
I20250905 07:55:45.672152 16642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:55:45.673060 16642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:55:45.674259 16642 hybrid_clock.cc:648] HybridClock initialized: now 1757058945674238 us; error 42 us; skew 500 ppm
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:55:45.678388 16642 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:55:45.679947 16642 webserver.cc:480] Webserver started at http://127.4.49.2:45175/ using document root <none> and password file <none>
I20250905 07:55:45.680702 16642 fs_manager.cc:362] Metadata directory not provided
I20250905 07:55:45.680822 16642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:55:45.681075 16642 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:45.696473 16642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "1e5601aff62c4d82b105b10254ad3cd6"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "71497b555ff41ecb4cd4f71f0d789161"
server_key_iv: "3ced31f5c7a2439b2b471a7d8fd07258"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.696980 16642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "1e5601aff62c4d82b105b10254ad3cd6"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "71497b555ff41ecb4cd4f71f0d789161"
server_key_iv: "3ced31f5c7a2439b2b471a7d8fd07258"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.701167 16642 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 07:55:45.717772 16658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:45.719242 16642 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250905 07:55:45.719405 16642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "1e5601aff62c4d82b105b10254ad3cd6"
format_stamp: "Formatted at 2025-09-05 07:55:45 on dist-test-slave-mnfb"
server_key: "71497b555ff41ecb4cd4f71f0d789161"
server_key_iv: "3ced31f5c7a2439b2b471a7d8fd07258"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:45.719672 16642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:55:45.751514 16642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:55:45.755834 16642 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:55:45.756146 16642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:55:45.757164 16642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:55:45.758473 16642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:55:45.758548 16642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:45.758602 16642 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:55:45.758682 16642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 07:55:45.796554 16642 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:37901
I20250905 07:55:45.798113 16642 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:55:45.805056 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 16642
I20250905 07:55:45.805182 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:55:45.805433 4292 external_mini_cluster.cc:1468] Setting key 5b63517f75de34e166fedd352752bb4b
I20250905 07:55:45.810971 16773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:37901 every 8 connection(s)
Sep 05 07:55:45 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:55:45.820564 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:45.801992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.2:50551 (local address 127.4.49.62:44209)
0905 07:55:45.803914 (+ 1922us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:45.803919 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:45.803941 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:55:45.806321 (+ 2380us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:45.806326 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:45.806387 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:45.806441 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:55:45.810653 (+ 4212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.811536 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:45.812615 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.812835 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:45.815925 (+ 3090us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:55:45.815955 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:55:45.815959 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:55:45.815994 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:55:45.817857 (+ 1863us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:45.818631 (+ 774us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:45.818636 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:45.818638 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:45.818703 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:45.819097 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:45.819101 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:45.819103 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:45.819318 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:55:45.819398 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:45.820203 (+ 805us) server_negotiation.cc:300] Negotiation successful
0905 07:55:45.820366 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805}
I20250905 07:55:45.822378 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:45.802821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44209 (local address 127.4.49.2:50551)
0905 07:55:45.803420 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:45.803472 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:45.804681 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:45.806947 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:45.806960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:45.809426 (+ 2466us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:55:45.810391 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:45.810414 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.811782 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:45.811786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:45.812443 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:45.812456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:45.813145 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:45.813893 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:55:45.813938 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:55:45.815692 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:55:45.818061 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:45.818068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:45.818085 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:45.818476 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:45.818817 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:45.818820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:45.818822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:45.818979 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:45.819470 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:55:45.819478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:55:45.819849 (+ 371us) client_negotiation.cc:769] Sending connection context
0905 07:55:45.821100 (+ 1251us) client_negotiation.cc:240] Negotiation successful
0905 07:55:45.821446 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20250905 07:55:45.824239 16774 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44209
I20250905 07:55:45.824540 16774 heartbeater.cc:461] Registering TS with master...
I20250905 07:55:45.825296 16774 heartbeater.cc:507] Master 127.4.49.62:44209 requested a full tablet report, sending...
I20250905 07:55:45.826727 16416 ts_manager.cc:194] Registered new tserver with Master: 1e5601aff62c4d82b105b10254ad3cd6 (127.4.49.2:37901)
I20250905 07:55:45.827601 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.2@KRBTEST.COM'} at 127.4.49.2:50551
WARNING: no policy specified for kudu/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:55:45.882575 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:43783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.3:0
--local_ip_for_outbound_sockets=127.4.49.3
--webserver_interface=127.4.49.3
--webserver_port=0
--tserver_master_addrs=127.4.49.62:44209
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:55:46.064844 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:55:46.065213 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:55:46.065330 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:55:46.070870 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:55:46.071004 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:55:46.071148 16781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.3
Sep 05 07:55:46 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:46.076989 16781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.49.20:43783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.49.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.3
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:55:46.078547 16781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:55:46.079700 16781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:55:46.089442 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:46.089751 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:55:46.090078 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:55:46.090374 16781 server_base.cc:1047] running on GCE node
I20250905 07:55:46.090925 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:55:46.091609 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:55:46.095336 16781 hybrid_clock.cc:648] HybridClock initialized: now 1757058946094858 us; error 487 us; skew 500 ppm
Sep 05 07:55:46 dist-test-slave-mnfb krb5kdc[15904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:55:46.099035 16781 init.cc:377] Logged in from keytab as kudu/127.4.49.3@KRBTEST.COM (short username kudu)
I20250905 07:55:46.101627 16781 webserver.cc:480] Webserver started at http://127.4.49.3:43861/ using document root <none> and password file <none>
I20250905 07:55:46.102437 16781 fs_manager.cc:362] Metadata directory not provided
I20250905 07:55:46.102547 16781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:55:46.102797 16781 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:55:46 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:55:46.131263 16781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data/instance:
uuid: "2c70451aadac43a384d2a055bc3c4d75"
format_stamp: "Formatted at 2025-09-05 07:55:46 on dist-test-slave-mnfb"
server_key: "bcb10f05da2b74db5adf3eaf28d1a795"
server_key_iv: "d5f750911475025483e11ad121da176b"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:46.132714 16781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance:
uuid: "2c70451aadac43a384d2a055bc3c4d75"
format_stamp: "Formatted at 2025-09-05 07:55:46 on dist-test-slave-mnfb"
server_key: "bcb10f05da2b74db5adf3eaf28d1a795"
server_key_iv: "d5f750911475025483e11ad121da176b"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:46.138410 16781 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 07:55:46.159927 16796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:46.161545 16781 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250905 07:55:46.161702 16781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal
uuid: "2c70451aadac43a384d2a055bc3c4d75"
format_stamp: "Formatted at 2025-09-05 07:55:46 on dist-test-slave-mnfb"
server_key: "bcb10f05da2b74db5adf3eaf28d1a795"
server_key_iv: "d5f750911475025483e11ad121da176b"
server_key_version: "kuduclusterkey@0"
I20250905 07:55:46.161837 16781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:55:46.179183 16781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:55:46.183106 16781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:55:46.183315 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:55:46.183979 16781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:55:46.185148 16781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:55:46.185216 16781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:46.185415 16781 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:55:46.185451 16781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:55:46.196913 16781 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.3:46495
I20250905 07:55:46.197250 16911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.3:46495 every 8 connection(s)
I20250905 07:55:46.198158 16781 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/data/info.pb
Sep 05 07:55:46 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.3@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:55:46.207878 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 16781
I20250905 07:55:46.208012 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal/instance
I20250905 07:55:46.208454 4292 external_mini_cluster.cc:1468] Setting key 969b252ff0015ef170f5148502fb8dbf
I20250905 07:55:46.211881 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.200005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.3:52461 (local address 127.4.49.62:44209)
0905 07:55:46.200173 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.200179 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.200998 (+ 819us) server_negotiation.cc:408] Connection header received
0905 07:55:46.201978 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.201983 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.202046 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.202121 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:55:46.203341 (+ 1220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.203886 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.204584 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.204773 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.206913 (+ 2140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:55:46.206937 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:55:46.206940 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:55:46.206970 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:55:46.208729 (+ 1759us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:46.209449 (+ 720us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:46.209452 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:46.209454 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:46.209515 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:46.210369 (+ 854us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:46.210373 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:46.210375 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:46.210593 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:55:46.210667 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.211523 (+ 856us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.211692 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250905 07:55:46.213152 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.200369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:44209 (local address 127.4.49.3:52461)
0905 07:55:46.200848 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.200895 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.201757 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.202296 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.202304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.202590 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:55:46.203175 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.203189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.204012 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.204015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.204456 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.204465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.204737 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.205383 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:55:46.205414 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:55:46.206719 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:55:46.208933 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:46.208941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:46.208957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:46.209335 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:46.210085 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:55:46.210088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:55:46.210089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:55:46.210259 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:55:46.210861 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:55:46.210868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:55:46.211212 (+ 344us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.212156 (+ 944us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.212453 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20250905 07:55:46.214288 16912 heartbeater.cc:344] Connected to a master server at 127.4.49.62:44209
I20250905 07:55:46.214561 16912 heartbeater.cc:461] Registering TS with master...
I20250905 07:55:46.215168 16912 heartbeater.cc:507] Master 127.4.49.62:44209 requested a full tablet report, sending...
I20250905 07:55:46.216291 16416 ts_manager.cc:194] Registered new tserver with Master: 2c70451aadac43a384d2a055bc3c4d75 (127.4.49.3:46495)
I20250905 07:55:46.216912 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.3@KRBTEST.COM'} at 127.4.49.3:52461
I20250905 07:55:46.222859 4292 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:55:46.231925 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.225036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.4.49.62:44209)
0905 07:55:46.225164 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.225168 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.225377 (+ 209us) server_negotiation.cc:408] Connection header received
0905 07:55:46.225506 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.225509 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.225567 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.225628 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:55:46.226453 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.227050 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.227790 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.227976 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.229075 (+ 1099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:55:46.229096 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:55:46.229098 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:55:46.229128 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:55:46.230476 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:46.230868 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:46.230871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:46.230872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:46.230919 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:55:46.231122 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:55:46.231125 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:55:46.231127 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:55:46.231284 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:55:46.231406 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.231620 (+ 214us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.231742 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250905 07:55:46.234688 16416 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:55:46.237326 16416 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:55:46.253827 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.247588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:37901 (local address 127.0.0.1:36040)
0905 07:55:46.248456 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.248465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.248513 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.250458 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.250461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.250479 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:55:46.250741 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.250750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.252590 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.252594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.253416 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.253425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.253527 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.253545 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.253591 (+ 46us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.253680 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":100,"threads_started":1}
I20250905 07:55:46.254348 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.246550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:46495 (local address 127.0.0.1:43128)
0905 07:55:46.248233 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.248245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.248308 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.249098 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.249104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.249126 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:55:46.249396 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.249406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.252147 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.252151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.253073 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.253084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.253215 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.253235 (+ 20us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.253283 (+ 48us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.253350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"thread_start_us":96,"threads_started":1}
I20250905 07:55:46.253021 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.246504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:46133 (local address 127.0.0.1:46972)
0905 07:55:46.248024 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.248098 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.248235 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.249763 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.249766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.249791 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:55:46.250016 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.250024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.251513 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.251517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.252514 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.252524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.252618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.252678 (+ 60us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.252762 (+ 84us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.252867 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"thread_start_us":151,"threads_started":1}
I20250905 07:55:46.255023 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.247763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.4.49.2:37901)
0905 07:55:46.250033 (+ 2270us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.250039 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.250061 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:55:46.250139 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.250144 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.250308 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.250888 (+ 580us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:55:46.251056 (+ 168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.252450 (+ 1394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.253913 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.254539 (+ 626us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.254679 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.254742 (+ 63us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.254844 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2169,"thread_start_us":82,"threads_started":1}
I20250905 07:55:46.256083 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.247360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.4.49.3:46495)
0905 07:55:46.248704 (+ 1344us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.248709 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.248730 (+ 21us) server_negotiation.cc:408] Connection header received
0905 07:55:46.248798 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.248802 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.248945 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.249071 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:55:46.249558 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.251735 (+ 2177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.253331 (+ 1596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.255601 (+ 2270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.255724 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.255792 (+ 68us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.255885 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":140,"threads_started":1}
I20250905 07:55:46.257086 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.246665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.4.49.1:46133)
0905 07:55:46.249379 (+ 2714us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.249385 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.249407 (+ 22us) server_negotiation.cc:408] Connection header received
0905 07:55:46.249491 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.249496 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.249647 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.249762 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:55:46.250155 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.251390 (+ 1235us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.254882 (+ 3492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.255462 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.255570 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.255634 (+ 64us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.255722 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2605,"thread_start_us":77,"threads_started":1}
I20250905 07:55:46.258193 16693 tablet_service.cc:1468] Processing CreateTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 (DEFAULT_TABLE table=test-table [id=d6c1a471864a4a0287e9fd90c212220b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:55:46.259301 16693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33efcf79d8f34e98ae8e2956efcc90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:55:46.259105 16845 tablet_service.cc:1468] Processing CreateTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 (DEFAULT_TABLE table=test-table [id=d6c1a471864a4a0287e9fd90c212220b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:55:46.260223 16845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33efcf79d8f34e98ae8e2956efcc90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:55:46.260406 16555 tablet_service.cc:1468] Processing CreateTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 (DEFAULT_TABLE table=test-table [id=d6c1a471864a4a0287e9fd90c212220b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:55:46.261400 16555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33efcf79d8f34e98ae8e2956efcc90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:55:46.269313 16930 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Bootstrap starting.
I20250905 07:55:46.270011 16932 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Bootstrap starting.
I20250905 07:55:46.270587 16931 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Bootstrap starting.
I20250905 07:55:46.273314 16932 tablet_bootstrap.cc:654] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Neither blocks nor log segments found. Creating new log.
I20250905 07:55:46.273478 16931 tablet_bootstrap.cc:654] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Neither blocks nor log segments found. Creating new log.
I20250905 07:55:46.274405 16930 tablet_bootstrap.cc:654] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Neither blocks nor log segments found. Creating new log.
I20250905 07:55:46.275010 16931 log.cc:826] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Log is configured to *not* fsync() on all Append() calls
I20250905 07:55:46.275010 16932 log.cc:826] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Log is configured to *not* fsync() on all Append() calls
I20250905 07:55:46.276387 16930 log.cc:826] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Log is configured to *not* fsync() on all Append() calls
I20250905 07:55:46.277443 16932 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: No bootstrap required, opened a new log
I20250905 07:55:46.277729 16932 ts_tablet_manager.cc:1397] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250905 07:55:46.277777 16931 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: No bootstrap required, opened a new log
I20250905 07:55:46.277953 16931 ts_tablet_manager.cc:1397] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20250905 07:55:46.278427 16930 tablet_bootstrap.cc:492] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: No bootstrap required, opened a new log
I20250905 07:55:46.278604 16930 ts_tablet_manager.cc:1397] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20250905 07:55:46.282874 16932 raft_consensus.cc:357] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.283165 16932 raft_consensus.cc:383] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:55:46.282868 16931 raft_consensus.cc:357] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.283239 16932 raft_consensus.cc:738] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bd3cbe15ce48159c3110bab7448117, State: Initialized, Role: FOLLOWER
I20250905 07:55:46.283326 16931 raft_consensus.cc:383] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:55:46.283520 16931 raft_consensus.cc:738] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5601aff62c4d82b105b10254ad3cd6, State: Initialized, Role: FOLLOWER
I20250905 07:55:46.283798 16932 consensus_queue.cc:260] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.284126 16931 consensus_queue.cc:260] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.284354 16930 raft_consensus.cc:357] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.284605 16930 raft_consensus.cc:383] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:55:46.284668 16930 raft_consensus.cc:738] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c70451aadac43a384d2a055bc3c4d75, State: Initialized, Role: FOLLOWER
I20250905 07:55:46.284994 16932 ts_tablet_manager.cc:1428] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 07:55:46.285149 16930 consensus_queue.cc:260] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.285418 16632 heartbeater.cc:499] Master 127.4.49.62:44209 was elected leader, sending a full tablet report...
I20250905 07:55:46.285389 16931 ts_tablet_manager.cc:1428] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250905 07:55:46.285789 16774 heartbeater.cc:499] Master 127.4.49.62:44209 was elected leader, sending a full tablet report...
I20250905 07:55:46.286185 16930 ts_tablet_manager.cc:1428] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20250905 07:55:46.286587 16912 heartbeater.cc:499] Master 127.4.49.62:44209 was elected leader, sending a full tablet report...
W20250905 07:55:46.310668 16775 tablet.cc:2378] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:55:46.402014 16633 tablet.cc:2378] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 05 07:55:46 dist-test-slave-mnfb krb5kdc[15904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058945, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
W20250905 07:55:46.449350 16913 tablet.cc:2378] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:55:46.495409 16936 raft_consensus.cc:491] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:55:46.495725 16936 raft_consensus.cc:513] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.497318 16936 leader_election.cc:290] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c70451aadac43a384d2a055bc3c4d75 (127.4.49.3:46495), 1e5601aff62c4d82b105b10254ad3cd6 (127.4.49.2:37901)
I20250905 07:55:46.507018 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.497991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:56163 (local address 127.4.49.3:46495)
0905 07:55:46.498135 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.498140 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.498380 (+ 240us) server_negotiation.cc:408] Connection header received
0905 07:55:46.498597 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.498602 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.498663 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.498717 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:55:46.499416 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.500363 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.505709 (+ 5346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.506438 (+ 729us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.506493 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.506774 (+ 281us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.506868 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250905 07:55:46.507764 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.497531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.3:46495 (local address 127.4.49.1:56163)
0905 07:55:46.498282 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.498298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.498464 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.498870 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.498873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.498896 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:55:46.499265 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.499277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.500513 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.500517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.501712 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.501724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.506593 (+ 4869us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.506615 (+ 22us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.507470 (+ 855us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.507569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":128,"threads_started":1}
I20250905 07:55:46.508857 16865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33efcf79d8f34e98ae8e2956efcc90b3" candidate_uuid: "06bd3cbe15ce48159c3110bab7448117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c70451aadac43a384d2a055bc3c4d75" is_pre_election: true
I20250905 07:55:46.509227 16865 raft_consensus.cc:2466] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bd3cbe15ce48159c3110bab7448117 in term 0.
I20250905 07:55:46.509615 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.498051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:52049 (local address 127.4.49.2:37901)
0905 07:55:46.498167 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.498171 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.502124 (+ 3953us) server_negotiation.cc:408] Connection header received
0905 07:55:46.502349 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.502353 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.502415 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.502467 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:55:46.503547 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.504413 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.508527 (+ 4114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.509108 (+ 581us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.509157 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.509392 (+ 235us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.509477 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250905 07:55:46.510054 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.497617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:37901 (local address 127.4.49.1:52049)
0905 07:55:46.502032 (+ 4415us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.502048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.502210 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.502620 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:55:46.502623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:55:46.502645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0905 07:55:46.503387 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.503399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.504601 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.504605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:55:46.505520 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:55:46.505531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.509231 (+ 3700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.509249 (+ 18us) client_negotiation.cc:769] Sending connection context
0905 07:55:46.509765 (+ 516us) client_negotiation.cc:240] Negotiation successful
0905 07:55:46.509848 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4299,"thread_start_us":147,"threads_started":1}
I20250905 07:55:46.510015 16517 leader_election.cc:304] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bd3cbe15ce48159c3110bab7448117, 2c70451aadac43a384d2a055bc3c4d75; no voters:
I20250905 07:55:46.510314 16936 raft_consensus.cc:2802] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:55:46.510397 16936 raft_consensus.cc:491] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:55:46.510442 16936 raft_consensus.cc:3058] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:55:46.511236 16726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33efcf79d8f34e98ae8e2956efcc90b3" candidate_uuid: "06bd3cbe15ce48159c3110bab7448117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5601aff62c4d82b105b10254ad3cd6" is_pre_election: true
I20250905 07:55:46.511559 16726 raft_consensus.cc:2466] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bd3cbe15ce48159c3110bab7448117 in term 0.
I20250905 07:55:46.512133 16936 raft_consensus.cc:513] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.512626 16936 leader_election.cc:290] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [CANDIDATE]: Term 1 election: Requested vote from peers 2c70451aadac43a384d2a055bc3c4d75 (127.4.49.3:46495), 1e5601aff62c4d82b105b10254ad3cd6 (127.4.49.2:37901)
I20250905 07:55:46.513042 16865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33efcf79d8f34e98ae8e2956efcc90b3" candidate_uuid: "06bd3cbe15ce48159c3110bab7448117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c70451aadac43a384d2a055bc3c4d75"
I20250905 07:55:46.513180 16865 raft_consensus.cc:3058] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:55:46.513309 16726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33efcf79d8f34e98ae8e2956efcc90b3" candidate_uuid: "06bd3cbe15ce48159c3110bab7448117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5601aff62c4d82b105b10254ad3cd6"
I20250905 07:55:46.513432 16726 raft_consensus.cc:3058] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:55:46.515532 16865 raft_consensus.cc:2466] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bd3cbe15ce48159c3110bab7448117 in term 1.
I20250905 07:55:46.515597 16726 raft_consensus.cc:2466] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bd3cbe15ce48159c3110bab7448117 in term 1.
I20250905 07:55:46.515982 16517 leader_election.cc:304] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bd3cbe15ce48159c3110bab7448117, 2c70451aadac43a384d2a055bc3c4d75; no voters:
I20250905 07:55:46.516330 16936 raft_consensus.cc:2802] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:55:46.516597 16936 raft_consensus.cc:695] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [term 1 LEADER]: Becoming Leader. State: Replica: 06bd3cbe15ce48159c3110bab7448117, State: Running, Role: LEADER
I20250905 07:55:46.516984 16936 consensus_queue.cc:237] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } }
I20250905 07:55:46.521258 16416 catalog_manager.cc:5582] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06bd3cbe15ce48159c3110bab7448117 (127.4.49.1). New cstate: current_term: 1 leader_uuid: "06bd3cbe15ce48159c3110bab7448117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bd3cbe15ce48159c3110bab7448117" member_type: VOTER last_known_addr { host: "127.4.49.1" port: 46133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 } health_report { overall_health: UNKNOWN } } }
I20250905 07:55:46.571425 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.566528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.4.49.1:46133)
0905 07:55:46.566683 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0905 07:55:46.566688 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0905 07:55:46.566707 (+ 19us) server_negotiation.cc:408] Connection header received
0905 07:55:46.566828 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:55:46.566832 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:55:46.566917 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:55:46.567050 (+ 133us) server_negotiation.cc:227] Negotiated authn=TOKEN
0905 07:55:46.567514 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.568457 (+ 943us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:55:46.569532 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:55:46.569823 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:55:46.569962 (+ 139us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0905 07:55:46.570590 (+ 628us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0905 07:55:46.570715 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0905 07:55:46.571207 (+ 492us) server_negotiation.cc:300] Negotiation successful
0905 07:55:46.571277 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250905 07:55:46.580139 16726 raft_consensus.cc:1273] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 1 FOLLOWER]: Refusing update from remote peer 06bd3cbe15ce48159c3110bab7448117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:55:46.580987 16936 consensus_queue.cc:1035] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5601aff62c4d82b105b10254ad3cd6" member_type: VOTER last_known_addr { host: "127.4.49.2" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:55:46.581112 16865 raft_consensus.cc:1273] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 1 FOLLOWER]: Refusing update from remote peer 06bd3cbe15ce48159c3110bab7448117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:55:46.581878 16936 consensus_queue.cc:1035] T 33efcf79d8f34e98ae8e2956efcc90b3 P 06bd3cbe15ce48159c3110bab7448117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c70451aadac43a384d2a055bc3c4d75" member_type: VOTER last_known_addr { host: "127.4.49.3" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:55:46.602596 16948 mvcc.cc:204] Tried to move back new op lower bound from 7196913445179506688 to 7196913444939452416. Current Snapshot: MvccSnapshot[applied={T|T < 7196913445179506688}]
I20250905 07:55:46.602679 16946 mvcc.cc:204] Tried to move back new op lower bound from 7196913445179506688 to 7196913444939452416. Current Snapshot: MvccSnapshot[applied={T|T < 7196913445179506688}]
I20250905 07:55:46.632920 16416 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:55:46.633118 16416 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20250905 07:55:46.636408 16416 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 00d0519ff7ad4be78fecf7c607c028f6: Sending DeleteTablet for 3 replicas of tablet 33efcf79d8f34e98ae8e2956efcc90b3
I20250905 07:55:46.637454 16555 tablet_service.cc:1515] Processing DeleteTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:55:46 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:46972
I20250905 07:55:46.638850 16693 tablet_service.cc:1515] Processing DeleteTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:55:46 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:36040
I20250905 07:55:46.639595 16845 tablet_service.cc:1515] Processing DeleteTablet for tablet 33efcf79d8f34e98ae8e2956efcc90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 07:55:46 UTC) from {username='kudu', principal='kudu/127.4.49.62@KRBTEST.COM'} at 127.0.0.1:43128
I20250905 07:55:46.640651 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 16499
I20250905 07:55:46.643922 16947 mvcc.cc:204] Tried to move back new op lower bound from 7196913445179506688 to 7196913444939452416. Current Snapshot: MvccSnapshot[applied={T|T < 7196913445179506688}]
I20250905 07:55:46.645028 16957 tablet_replica.cc:331] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6: stopping tablet replica
I20250905 07:55:46.645370 16957 raft_consensus.cc:2241] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:55:46.645716 16957 raft_consensus.cc:2270] T 33efcf79d8f34e98ae8e2956efcc90b3 P 1e5601aff62c4d82b105b10254ad3cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:55:46.646203 16958 tablet_replica.cc:331] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: stopping tablet replica
I20250905 07:55:46.646564 16958 raft_consensus.cc:2241] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:55:46.647006 16958 raft_consensus.cc:2270] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:55:46.648708 16958 ts_tablet_manager.cc:1905] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 07:55:46.651664 16402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.49.1:46133 (error 108)
I20250905 07:55:46.651835 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 16642
I20250905 07:55:46.654513 16958 ts_tablet_manager.cc:1918] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 07:55:46.654770 16958 log.cc:1199] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Deleting WAL directory at /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1757058797595744-4292-0/minicluster-data/ts-2/wal/wals/33efcf79d8f34e98ae8e2956efcc90b3
I20250905 07:55:46.655236 16958 ts_tablet_manager.cc:1939] T 33efcf79d8f34e98ae8e2956efcc90b3 P 2c70451aadac43a384d2a055bc3c4d75: Deleting consensus metadata
I20250905 07:55:46.656772 16400 catalog_manager.cc:4928] TS 2c70451aadac43a384d2a055bc3c4d75 (127.4.49.3:46495): tablet 33efcf79d8f34e98ae8e2956efcc90b3 (table test-table [id=d6c1a471864a4a0287e9fd90c212220b]) successfully deleted
I20250905 07:55:46.657632 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:55:46.657225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.1:46133 (local address 127.0.0.1:46984)
0905 07:55:46.657422 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.657528 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.1:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20250905 07:55:46.657924 16402 catalog_manager.cc:4671] TS 06bd3cbe15ce48159c3110bab7448117 (127.4.49.1:46133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33efcf79d8f34e98ae8e2956efcc90b3: Network error: Client connection negotiation failed: client connection to 127.4.49.1:46133: connect: Connection refused (error 111)
W20250905 07:55:46.664548 16928 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:55:46.663834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.2:37901 (local address 127.0.0.1:36042)
0905 07:55:46.664075 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0905 07:55:46.664092 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0905 07:55:46.664199 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:55:46.664418 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.2:37901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130}
I20250905 07:55:46.664717 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 16781
I20250905 07:55:46.671478 4292 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task5YGlfE/build/debug/bin/kudu with pid 16383
I20250905 07:55:46.679593 4292 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20250905 07:55:47.013965 4292 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20250905 07:55:47.014111 4292 mini_ranger.cc:66] Stopping Ranger...
I20250905 07:55:47.349392 4292 mini_ranger.cc:68] Stopped Ranger
2025-09-05 07:55:47.349 UTC [15936] LOG: received smart shutdown request
2025-09-05 07:55:47.352 UTC [15936] LOG: background worker "logical replication launcher" (PID 15944) exited with exit code 1
2025-09-05 07:55:47.353 UTC [15939] LOG: shutting down
2025-09-05 07:55:47.355 UTC [15939] LOG: checkpoint starting: shutdown immediate
2025-09-05 07:55:48.190 UTC [15939] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.035 s, sync=0.794 s, total=0.838 s; sync files=1108, longest=0.009 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045180, redo lsn=0/2045180
2025-09-05 07:55:48.197 UTC [15936] LOG: database system is shut down
2025-09-05T07:55:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54417 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 05 07:55:48 dist-test-slave-mnfb krb5kdc[16968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 05 07:55:48 dist-test-slave-mnfb krb5kdc[16968](info): set up 2 sockets
Sep 05 07:55:48 dist-test-slave-mnfb krb5kdc[16968](info): commencing operation
krb5kdc: starting...
W20250905 07:55:50.126758 4292 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/test-user.keytab.
Sep 05 07:55:50 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-05T07:55:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:55:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangeradmin_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangeradmin_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangerlookup_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangerlookup_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
I20250905 07:55:50.288916 4292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5YGlfE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/postgres -l logfile start
2025-09-05 07:55:52.599 UTC [17000] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-05 07:55:52.599 UTC [17000] LOG: listening on IPv4 address "127.4.49.20", port 45067
2025-09-05 07:55:52.602 UTC [17000] LOG: listening on Unix socket "/tmp/.s.PGSQL.45067"
2025-09-05 07:55:52.607 UTC [17005] LOG: database system was shut down at 2025-09-05 07:55:51 UTC
2025-09-05 07:55:52.612 UTC [17000] LOG: database system is ready to accept connections
I20250905 07:55:54.370186 4292 mini_postgres.cc:96] Postgres bound to 45067
2025-09-05 07:55:54.375 UTC [17011] FATAL: database "slave" does not exist
127.4.49.20:45067 - accepting connections
I20250905 07:55:54.376447 4292 mini_ranger.cc:161] Starting Ranger...
I20250905 07:55:54.394488 4292 mini_ranger.cc:84] Created miniranger Postgres user
I20250905 07:55:54.457569 4292 mini_ranger.cc:87] Created ranger Postgres database
I20250905 07:55:54.457651 4292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin
2025-09-05 07:55:54,749 [I] DB FLAVOR :POSTGRES
2025-09-05 07:55:54,749 [I] --------- Verifying Ranger DB connection ---------
2025-09-05 07:55:54,750 [I] Checking connection..
2025-09-05 07:55:54,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-05 07:55:55,067 [I] Checking connection passed.
2025-09-05 07:55:55,067 [I] --------- Verifying version history table ---------
2025-09-05 07:55:55,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:55,416 [I] Table x_db_version_h does not exist in database ranger
2025-09-05 07:55:55,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:55,757 [I] Table x_db_version_h does not exist in database ranger
2025-09-05 07:55:55,758 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-05 07:55:55,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-05 07:55:56.080 UTC [17080] WARNING: there is no transaction in progress
2025-09-05 07:55:56,087 [I] create_dbversion_catalog.sql file imported successfully
2025-09-05 07:55:56,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-05 07:55:56,411 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-05 07:55:56,411 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-05 07:55:56,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-05 07:55:56,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-05 07:55:57,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mnfb.c.gcp-upstream.internal','N') ;"
2025-09-05 07:55:57,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-05 07:55:57,710 [I] Table x_portal_user does not exist in database ranger
2025-09-05 07:55:57,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-05 07:55:58,050 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-05 07:55:58,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-05 07:55:58,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-05 07:55:58,674 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-05 07:55:58,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-05 07:55:59.524 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.535 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.547 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.558 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.570 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.582 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.627 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.632 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.642 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.652 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.662 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.671 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.679 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.684 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.688 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:55:59.691 UTC [17199] WARNING: there is no transaction in progress
2025-09-05 07:56:00,098 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-05 07:56:00,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-05 07:56:00,411 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-05 07:56:00,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-05 07:56:00,743 [I] Table x_portal_user already exists in database 'ranger'
2025-09-05 07:56:00,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-05 07:56:01,083 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-05 07:56:01,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-05 07:56:01,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-05 07:56:01,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mnfb.c.gcp-upstream.internal';"
2025-09-05 07:56:02,029 [I] CORE_DB_SCHEMA import status has been updated
2025-09-05 07:56:02,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-05 07:56:02,345 [I] DB_PATCHES have already been applied
I20250905 07:56:02.351478 4292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5YGlfE/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250905 07:56:02.351593 4292 mini_ranger.cc:193] Using host: 127.4.49.20
I20250905 07:56:02.354492 4292 mini_ranger.cc:227] Ranger admin URL: http://127.4.49.20:51995
Sep 05, 2025 7:56:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 05, 2025 7:56:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 05, 2025 7:56:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp
Sep 05, 2025 7:56:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 05, 2025 7:56:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 05, 2025 7:56:02 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 05, 2025 7:56:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-admin/ews/webapp].
Sep 05, 2025 7:56:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 05, 2025 7:56:03 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-51995"]
Sep 05, 2025 7:56:03 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 05, 2025 7:56:03 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 05, 2025 7:56:03 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 05, 2025 7:56:03 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 05, 2025 7:56:03 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 05, 2025 7:56:04 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 05, 2025 7:56:04 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 05, 2025 7:56:04 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Sep 05, 2025 7:56:04 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20250905 07:56:04.414338 4292 mini_ranger.cc:160] Time spent starting Ranger: real 10.038s user 0.002s sys 0.007s
Sep 05, 2025 7:56:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 05, 2025 7:56:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 05, 2025 7:56:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 05, 2025 7:56:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 05, 2025 7:56:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 05, 2025 7:56:25 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-51995"]
I20250905 07:56:25.706966 4292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangerkms_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/rangerkms_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.49.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.49.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.49.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/HTTP_127.4.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 05 07:56:25 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250905 07:56:25.778898 4292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250905 07:56:25.788189 4292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250905 07:56:25.884729 4292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250905 07:56:25.884816 4292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms
I20250905 07:56:25.884833 4292 mini_ranger_kms.cc:227] Using postgres at 127.4.49.20:45067
2025-09-05 07:56:26,050 [I] DB FLAVOR :POSTGRES
2025-09-05 07:56:26,050 [I] --------- Verifying Ranger DB connection ---------
2025-09-05 07:56:26,050 [I] Checking connection
2025-09-05 07:56:26,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-05 07:56:26,389 [I] connection success
2025-09-05 07:56:26,389 [I] --------- Verifying Ranger DB tables ---------
2025-09-05 07:56:26,389 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-05 07:56:26,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-05 07:56:26,718 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-05 07:56:26,718 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-05 07:56:26,719 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-05 07:56:26,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.49.20:45067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-05 07:56:27,077 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250905 07:56:27.580845 4292 mini_ranger_kms.cc:310] Created kms service
I20250905 07:56:27.798373 4292 mini_ranger_kms.cc:326] Created kudu user
I20250905 07:56:27.842069 4292 mini_ranger_kms.cc:343] Created rangerkms user
Sep 05 07:56:27 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:56:28.244236 4292 mini_ranger_kms.cc:384] Added ranger policy
I20250905 07:56:28.244472 4292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms:/tmp/dist-test-task5YGlfE/build/debug/bin/postgresql.jar:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5YGlfE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250905 07:56:28.244513 4292 mini_ranger_kms.cc:240] Using host: 127.4.49.20
I20250905 07:56:28.247758 4292 mini_ranger_kms.cc:276] Ranger KMS PID: 17394
I20250905 07:56:28.247859 4292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.49.20:35673
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-kms/ews/webapp].
Sep 05, 2025 7:56:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 05, 2025 7:56:29 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-35673"]
Sep 05, 2025 7:56:29 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 05, 2025 7:56:29 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 05, 2025 7:56:30 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20250905 07:56:30.305187 4292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.526s user 0.009s sys 0.138s
I20250905 07:56:30.305311 4292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250905 07:56:30.305389 4292 mini_ranger_kms.cc:401] 127.4.49.20:35673/kms/v1/keys
Sep 05, 2025 7:56:30 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 05 07:56:30 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1757058990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 05 07:56:30 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
Sep 05 07:56:31 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.49.20@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
Sep 05 07:56:33 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:56:33.644104 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38907
--webserver_interface=127.4.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.49.62:38907
--ranger_config_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:35673/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:56:33.753212 17448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:56:33.753463 17448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:56:33.753520 17448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:56:33.756929 17448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:56:33.757010 17448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:56:33.757032 17448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:56:33.757050 17448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:56:33.757067 17448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:56:33.761901 17448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.49.20:35673/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.49.62:38907
--ranger_config_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.49.62:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.49.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:56:33.763038 17448 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:56:33.763929 17448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:56:33.769692 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:56:33.769857 17448 server_base.cc:1047] running on GCE node
W20250905 07:56:33.769681 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:56:33.769701 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:56:33.770561 17448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:56:33.771466 17448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:56:33.772632 17448 hybrid_clock.cc:648] HybridClock initialized: now 1757058993772612 us; error 39 us; skew 500 ppm
Sep 05 07:56:33 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:56:33.775247 17448 init.cc:377] Logged in from keytab as kudu/127.4.49.62@KRBTEST.COM (short username kudu)
I20250905 07:56:33.776366 17448 webserver.cc:480] Webserver started at http://127.4.49.62:46331/ using document root <none> and password file <none>
I20250905 07:56:33.776913 17448 fs_manager.cc:362] Metadata directory not provided
I20250905 07:56:33.776963 17448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:56:33.777141 17448 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:56:33 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.62@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:56:33.831923 17448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data/instance:
uuid: "c11d32f815704bbaae8d1f518b39e034"
format_stamp: "Formatted at 2025-09-05 07:56:33 on dist-test-slave-mnfb"
server_key: "261c6bad11f1edf0a0c342e9bab23eea"
server_key_iv: "99e6e4f10ab760f6ff8aac2d3740ffbe"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:33.832607 17448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal/instance:
uuid: "c11d32f815704bbaae8d1f518b39e034"
format_stamp: "Formatted at 2025-09-05 07:56:33 on dist-test-slave-mnfb"
server_key: "261c6bad11f1edf0a0c342e9bab23eea"
server_key_iv: "99e6e4f10ab760f6ff8aac2d3740ffbe"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:33.837249 17448 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 07:56:33.854707 17463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:33.856189 17448 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:56:33.856343 17448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal
uuid: "c11d32f815704bbaae8d1f518b39e034"
format_stamp: "Formatted at 2025-09-05 07:56:33 on dist-test-slave-mnfb"
server_key: "261c6bad11f1edf0a0c342e9bab23eea"
server_key_iv: "99e6e4f10ab760f6ff8aac2d3740ffbe"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:33.856475 17448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:56:33.877532 17448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:56:33.884548 17448 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:56:33.884771 17448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:56:33.893388 17448 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.62:38907
I20250905 07:56:33.893388 17515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.62:38907 every 8 connection(s)
I20250905 07:56:33.894658 17448 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/data/info.pb
I20250905 07:56:33.897675 17516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:56:33.901773 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 17448
I20250905 07:56:33.901921 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/wal/instance
I20250905 07:56:33.902156 4292 external_mini_cluster.cc:1468] Setting key 0c3641873bdbc7da8ae968c3909814c0
I20250905 07:56:33.903241 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034: Bootstrap starting.
I20250905 07:56:33.906066 17516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034: Neither blocks nor log segments found. Creating new log.
I20250905 07:56:33.906924 17516 log.cc:826] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034: Log is configured to *not* fsync() on all Append() calls
I20250905 07:56:33.909204 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034: No bootstrap required, opened a new log
Sep 05 07:56:33 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:56:33.912844 17516 raft_consensus.cc:357] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } }
I20250905 07:56:33.913108 17516 raft_consensus.cc:383] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:56:33.913201 17516 raft_consensus.cc:738] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11d32f815704bbaae8d1f518b39e034, State: Initialized, Role: FOLLOWER
I20250905 07:56:33.913759 17516 consensus_queue.cc:260] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } }
I20250905 07:56:33.913909 17516 raft_consensus.cc:397] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 07:56:33.913990 17516 raft_consensus.cc:491] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 07:56:33.914110 17516 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:56:33.915088 17516 raft_consensus.cc:513] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } }
I20250905 07:56:33.915629 17516 leader_election.cc:304] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c11d32f815704bbaae8d1f518b39e034; no voters:
I20250905 07:56:33.915959 17516 leader_election.cc:290] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 07:56:33.916085 17522 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:56:33.916285 17522 raft_consensus.cc:695] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [term 1 LEADER]: Becoming Leader. State: Replica: c11d32f815704bbaae8d1f518b39e034, State: Running, Role: LEADER
I20250905 07:56:33.916617 17522 consensus_queue.cc:237] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } }
I20250905 07:56:33.916769 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:33.903488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.4.49.62:38907)
0905 07:56:33.903953 (+ 465us) server_negotiation.cc:207] Beginning negotiation
0905 07:56:33.903965 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0905 07:56:33.904005 (+ 40us) server_negotiation.cc:408] Connection header received
0905 07:56:33.904767 (+ 762us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:56:33.904793 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:56:33.905183 (+ 390us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:56:33.905542 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:56:33.906566 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:56:33.907580 (+ 1014us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:56:33.908469 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:56:33.909104 (+ 635us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:56:33.911292 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:56:33.911319 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:56:33.911335 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:56:33.911382 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:56:33.913467 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:56:33.914064 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:56:33.914070 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:56:33.914077 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:56:33.914177 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:56:33.914460 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:56:33.914463 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:56:33.914465 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:56:33.914944 (+ 479us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:56:33.915148 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0905 07:56:33.915454 (+ 306us) server_negotiation.cc:300] Negotiation successful
0905 07:56:33.915896 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":153,"threads_started":1}
I20250905 07:56:33.917371 17516 sys_catalog.cc:564] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:56:33.917867 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c11d32f815704bbaae8d1f518b39e034. Latest consensus state: current_term: 1 leader_uuid: "c11d32f815704bbaae8d1f518b39e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } } }
I20250905 07:56:33.918020 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [sys.catalog]: This master's current role is: LEADER
I20250905 07:56:33.918224 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c11d32f815704bbaae8d1f518b39e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d32f815704bbaae8d1f518b39e034" member_type: VOTER last_known_addr { host: "127.4.49.62" port: 38907 } } }
I20250905 07:56:33.918322 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034 [sys.catalog]: This master's current role is: LEADER
I20250905 07:56:33.919829 17516 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250905 07:56:33.921041 17526 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:56:33.927174 17526 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:56:33.935006 17526 catalog_manager.cc:1349] Generated new cluster ID: 9f602e01be4d4555af290678e1141cb9
I20250905 07:56:33.935192 17526 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:56:33.948848 17526 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:56:33.950275 17526 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:56:33.961302 17526 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c11d32f815704bbaae8d1f518b39e034: Generated new TSK 0
I20250905 07:56:33.962423 17526 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:56:34.704514 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:35673/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.1:0
--local_ip_for_outbound_sockets=127.4.49.1
--webserver_interface=127.4.49.1
--webserver_port=0
--tserver_master_addrs=127.4.49.62:38907
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250905 07:56:34.819006 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:56:34.819273 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:56:34.819363 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:56:34.822777 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:56:34.822852 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:56:34.822966 17564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.1
I20250905 07:56:34.827773 17564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.49.20:35673/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.49.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.1
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:56:34.828956 17564 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:56:34.829936 17564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:56:34.837761 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:56:34.838250 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:56:34.839084 17564 server_base.cc:1047] running on GCE node
W20250905 07:56:34.839111 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:56:34.839520 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:56:34.840127 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:56:34.841276 17564 hybrid_clock.cc:648] HybridClock initialized: now 1757058994841243 us; error 51 us; skew 500 ppm
Sep 05 07:56:34 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:56:34.843773 17564 init.cc:377] Logged in from keytab as kudu/127.4.49.1@KRBTEST.COM (short username kudu)
I20250905 07:56:34.844858 17564 webserver.cc:480] Webserver started at http://127.4.49.1:38591/ using document root <none> and password file <none>
I20250905 07:56:34.845395 17564 fs_manager.cc:362] Metadata directory not provided
I20250905 07:56:34.845443 17564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:56:34.845618 17564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:56:34 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:56:34.860509 17564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data/instance:
uuid: "7681d6d99283442ca0dd7a0856bef629"
format_stamp: "Formatted at 2025-09-05 07:56:34 on dist-test-slave-mnfb"
server_key: "0b7de09eb1370583c4efbfc533018a6a"
server_key_iv: "5242308db1b0f3afe873a295942cf0ee"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:34.861193 17564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance:
uuid: "7681d6d99283442ca0dd7a0856bef629"
format_stamp: "Formatted at 2025-09-05 07:56:34 on dist-test-slave-mnfb"
server_key: "0b7de09eb1370583c4efbfc533018a6a"
server_key_iv: "5242308db1b0f3afe873a295942cf0ee"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:34.866071 17564 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 07:56:34.894387 17581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.002s user 0.000s sys 0.000s
I20250905 07:56:34.896155 17564 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20250905 07:56:34.896322 17564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal
uuid: "7681d6d99283442ca0dd7a0856bef629"
format_stamp: "Formatted at 2025-09-05 07:56:34 on dist-test-slave-mnfb"
server_key: "0b7de09eb1370583c4efbfc533018a6a"
server_key_iv: "5242308db1b0f3afe873a295942cf0ee"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:34.896437 17564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:56:34.947486 17564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:56:34.954337 17564 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:56:34.954632 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:56:34.955372 17564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:56:34.956558 17564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:56:34.956661 17564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:34.956754 17564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:56:34.956807 17564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:34.967741 17564 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.1:38787
I20250905 07:56:34.969336 17564 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/data/info.pb
I20250905 07:56:34.974012 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 17564
I20250905 07:56:34.974150 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-0/wal/instance
I20250905 07:56:34.974406 4292 external_mini_cluster.cc:1468] Setting key 2157cab49b1d2fa9eec595ef192ba040
I20250905 07:56:34.970996 17697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.1:38787 every 8 connection(s)
Sep 05 07:56:34 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.1@KRBTEST.COM for kudu/127.4.49.62@KRBTEST.COM
I20250905 07:56:34.987468 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:34.973002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.49.1:48355 (local address 127.4.49.62:38907)
0905 07:56:34.976161 (+ 3159us) server_negotiation.cc:207] Beginning negotiation
0905 07:56:34.976165 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0905 07:56:34.976179 (+ 14us) server_negotiation.cc:408] Connection header received
0905 07:56:34.976217 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:56:34.976220 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:56:34.976270 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:56:34.976349 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0905 07:56:34.978050 (+ 1701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:56:34.978880 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0905 07:56:34.980487 (+ 1607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0905 07:56:34.980747 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:56:34.983170 (+ 2423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0905 07:56:34.983197 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0905 07:56:34.983201 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0905 07:56:34.983233 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0905 07:56:34.985111 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:56:34.985736 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:56:34.985740 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:56:34.985742 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:56:34.985796 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0905 07:56:34.986128 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0905 07:56:34.986131 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0905 07:56:34.986133 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0905 07:56:34.986333 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0905 07:56:34.986437 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0905 07:56:34.987118 (+ 681us) server_negotiation.cc:300] Negotiation successful
0905 07:56:34.987258 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3070,"thread_start_us":78,"threads_started":1}
I20250905 07:56:34.988471 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:34.973386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:48355)
0905 07:56:34.974527 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:34.974589 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0905 07:56:34.975705 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:56:34.976659 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0905 07:56:34.976672 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0905 07:56:34.977039 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0905 07:56:34.977871 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:56:34.977891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:56:34.979040 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0905 07:56:34.979044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0905 07:56:34.979677 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0905 07:56:34.979688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0905 07:56:34.979922 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0905 07:56:34.981165 (+ 1243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0905 07:56:34.981202 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0905 07:56:34.982975 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0905 07:56:34.985259 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:56:34.985266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:56:34.985281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:56:34.985625 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:56:34.985892 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0905 07:56:34.985896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0905 07:56:34.985898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0905 07:56:34.986033 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0905 07:56:34.986450 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0905 07:56:34.986458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0905 07:56:34.986796 (+ 338us) client_negotiation.cc:769] Sending connection context
0905 07:56:34.987068 (+ 272us) client_negotiation.cc:240] Negotiation successful
0905 07:56:34.987365 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":118,"threads_started":1}
I20250905 07:56:34.989923 17698 heartbeater.cc:344] Connected to a master server at 127.4.49.62:38907
I20250905 07:56:34.990216 17698 heartbeater.cc:461] Registering TS with master...
I20250905 07:56:34.990942 17698 heartbeater.cc:507] Master 127.4.49.62:38907 requested a full tablet report, sending...
I20250905 07:56:34.992944 17479 ts_manager.cc:194] Registered new tserver with Master: 7681d6d99283442ca0dd7a0856bef629 (127.4.49.1:38787)
I20250905 07:56:34.994616 17479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.49.1@KRBTEST.COM'} at 127.4.49.1:48355
WARNING: no policy specified for kudu/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab.
I20250905 07:56:35.032116 4292 external_mini_cluster.cc:1366] Running /tmp/dist-test-task5YGlfE/build/debug/bin/kudu
/tmp/dist-test-task5YGlfE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.49.20:35673/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.49.2:0
--local_ip_for_outbound_sockets=127.4.49.2
--webserver_interface=127.4.49.2
--webserver_port=0
--tserver_master_addrs=127.4.49.62:38907
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.49.20:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.4.49.62 from keytab /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20250905 07:56:35.050961 17556 server.cc:273] Received an EOF from the subprocess
F20250905 07:56:35.051929 17541 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f0cc9bfddcd google::LogMessage::Fail() at ??:0
@ 0x7f0cc9c01b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0cc9bfd7cc google::LogMessage::Flush() at ??:0
@ 0x7f0cc9bfef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0ccb8202d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f0ccb81d0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f0ccb821005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f21340a370 std::function<>::operator()() at ??:0
@ 0x7f0ccb053924 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0ccb85c6db start_thread at ??:0
@ 0x7f0cc8db471f clone at ??:0
W20250905 07:56:35.151134 17706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:56:35.151401 17706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:56:35.151490 17706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:56:35.155642 17706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250905 07:56:35.155737 17706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:56:35.155867 17706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.49.2
I20250905 07:56:35.160940 17706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.49.20:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.49.20:35673/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.49.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.49.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.49.2
--log_dir=/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type DEBUG
built by None at 05 Sep 2025 07:43:12 UTC on 24a791456cd2
build id 7906
I20250905 07:56:35.162111 17706 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:56:35.163089 17706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:56:35.169714 17724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:56:35.169709 17723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:56:35.169703 17726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:56:35.170130 17706 server_base.cc:1047] running on GCE node
I20250905 07:56:35.170501 17706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:56:35.171059 17706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:56:35.172247 17706 hybrid_clock.cc:648] HybridClock initialized: now 1757058995172231 us; error 35 us; skew 500 ppm
Sep 05 07:56:35 dist-test-slave-mnfb krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058995, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250905 07:56:35.174823 17706 init.cc:377] Logged in from keytab as kudu/127.4.49.2@KRBTEST.COM (short username kudu)
I20250905 07:56:35.175988 17706 webserver.cc:480] Webserver started at http://127.4.49.2:40007/ using document root <none> and password file <none>
I20250905 07:56:35.176573 17706 fs_manager.cc:362] Metadata directory not provided
I20250905 07:56:35.176630 17706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:56:35.176810 17706 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 05 07:56:35 dist-test-slave-mnfb krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1757058995, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.49.2@KRBTEST.COM for HTTP/127.4.49.20@KRBTEST.COM
I20250905 07:56:35.192687 17706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data/instance:
uuid: "6271c43d60e04da9b5ce42f3ae246442"
format_stamp: "Formatted at 2025-09-05 07:56:35 on dist-test-slave-mnfb"
server_key: "ebb1358dc8484331cbee981af475efbe"
server_key_iv: "8aa093807a9ac417f3db29935aa32529"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:35.193428 17706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance:
uuid: "6271c43d60e04da9b5ce42f3ae246442"
format_stamp: "Formatted at 2025-09-05 07:56:35 on dist-test-slave-mnfb"
server_key: "ebb1358dc8484331cbee981af475efbe"
server_key_iv: "8aa093807a9ac417f3db29935aa32529"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:35.198441 17706 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250905 07:56:35.216405 17734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:35.217893 17706 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:56:35.218130 17706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data,/tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal
uuid: "6271c43d60e04da9b5ce42f3ae246442"
format_stamp: "Formatted at 2025-09-05 07:56:35 on dist-test-slave-mnfb"
server_key: "ebb1358dc8484331cbee981af475efbe"
server_key_iv: "8aa093807a9ac417f3db29935aa32529"
server_key_version: "kuduclusterkey@0"
I20250905 07:56:35.218349 17706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:56:35.248998 17706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:56:35.252920 17706 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:56:35.253196 17706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:56:35.253963 17706 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:56:35.255175 17706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:56:35.255285 17706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:35.255378 17706 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:56:35.255442 17706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:56:35.267159 17706 rpc_server.cc:307] RPC server started. Bound to: 127.4.49.2:46775
I20250905 07:56:35.267418 17849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.49.2:46775 every 8 connection(s)
I20250905 07:56:35.268455 17706 server_base.cc:1179] Dumped server information to /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/data/info.pb
I20250905 07:56:35.269373 4292 external_mini_cluster.cc:1428] Started /tmp/dist-test-task5YGlfE/build/debug/bin/kudu as pid 17706
I20250905 07:56:35.269484 4292 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task5YGlfE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1757058797595744-4292-0/minicluster-data/ts-1/wal/instance
I20250905 07:56:35.269752 4292 external_mini_cluster.cc:1468] Setting key c19b1fa7e262691be1c4b230de5fc594
I20250905 07:56:35.272300 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:35.270972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:44213)
0905 07:56:35.271490 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:35.271590 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":151,"threads_started":1}
W20250905 07:56:35.272773 17850 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:38907 (0 consecutive failures): Network error: Failed to ping master at 127.4.49.62:38907: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
I20250905 07:56:35.273442 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:35.273225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:38965)
0905 07:56:35.273335 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:35.273379 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20250905 07:56:35.274050 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:35.273869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:58167)
0905 07:56:35.273960 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:35.273999 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20250905 07:56:35.274194 17850 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250905 07:56:35.997951 17698 heartbeater.cc:499] Master 127.4.49.62:38907 was elected leader, sending a full tablet report...
I20250905 07:56:35.998936 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:35.998460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:54271)
0905 07:56:35.998747 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:35.998824 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20250905 07:56:35.999212 17698 heartbeater.cc:646] Failed to heartbeat to 127.4.49.62:38907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
I20250905 07:56:36.275231 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:36.274791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:32815)
0905 07:56:36.275067 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:36.275126 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20250905 07:56:37.000394 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:36.999923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:39373)
0905 07:56:37.000226 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:37.000291 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20250905 07:56:37.276669 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:37.276156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:50749)
0905 07:56:37.276485 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:37.276549 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":81,"threads_started":1}
I20250905 07:56:38.001787 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:38.001257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:33123)
0905 07:56:38.001586 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:38.001657 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20250905 07:56:38.277900 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:38.277442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:52167)
0905 07:56:38.277721 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:38.277785 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20250905 07:56:39.003028 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:39.002606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:55943)
0905 07:56:39.002879 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:39.002932 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20250905 07:56:39.279181 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:39.278702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:47453)
0905 07:56:39.279016 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:39.279081 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20250905 07:56:40.004247 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:40.003790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.1:39003)
0905 07:56:40.004074 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:40.004142 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20250905 07:56:40.280668 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0905 07:56:40.280216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.49.62:38907 (local address 127.4.49.2:38649)
0905 07:56:40.280510 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0905 07:56:40.280568 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.49.62:38907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
F20250905 07:56:40.310441 4292 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1757059000 (unix time) try "date -d @1757059000" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000010c4) received by PID 4292 (TID 0x7f6f22fcbd80) from PID 4292; stack trace: ***
@ 0x7f6f2eb0a980 (unknown) at ??:0
@ 0x7f6f2c1d5fb7 gsignal at ??:0
@ 0x7f6f2c1d7921 abort at ??:0
@ 0x7f6f2d443dcd google::LogMessage::Fail() at ??:0
@ 0x7f6f2d447b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6f2d4437cc google::LogMessage::Flush() at ??:0
@ 0x7f6f2d444f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f300dbcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f6f300d89c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5571d2702919 kudu::SecurityITest::StartCluster() at ??:0
@ 0x5571d26eac15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f6f2e44e0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f6f2e442bdb testing::Test::Run() at ??:0
@ 0x7f6f2e442d9d testing::TestInfo::Run() at ??:0
@ 0x7f6f2e443377 testing::TestSuite::Run() at ??:0
@ 0x7f6f2e44377c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f6f2e44e60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f6f2e442e63 testing::UnitTest::Run() at ??:0
@ 0x5571d273ac58 RUN_ALL_TESTS() at ??:0
@ 0x5571d2739e45 main at ??:0
@ 0x7f6f2c1b8bf7 __libc_start_main at ??:0
@ 0x5571d26d206a _start at ??:0
2025-09-05 07:56:40.490 UTC [17426] FATAL: terminating connection due to unexpected postmaster exit
2025-09-05 07:56:40.490 UTC [17318] FATAL: terminating connection due to unexpected postmaster exit
2025-09-05 07:56:40.491 UTC [17321] FATAL: terminating connection due to unexpected postmaster exit
2025-09-05 07:56:40.492 UTC [17317] FATAL: terminating connection due to unexpected postmaster exit
2025-09-05 07:56:40.492 UTC [17320] FATAL: terminating connection due to unexpected postmaster exit
2025-09-05 07:56:40.493 UTC [17316] FATAL: terminating connection due to unexpected postmaster exit