[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:25 dist-test-slave-4wpb krb5kdc[19725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:25 dist-test-slave-4wpb krb5kdc[19725](info): set up 2 sockets
Sep 03 14:26:25 dist-test-slave-4wpb krb5kdc[19725](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250903 14:26:27.705384 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.846s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:27 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:27.822088 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39533
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:27.913115 19741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:27.913280 19741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:27.913352 19741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:27.916163 19741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:27.916213 19741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:27.916247 19741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:27.916275 19741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:27.916307 19741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:27.919816 19741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:27.920648 19741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:27.921362 19741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:27.926182 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:27.926187 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:27.926182 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:27.926311 19741 server_base.cc:1047] running on GCE node
I20250903 14:26:27.926780 19741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:27.927497 19741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:27.928629 19741 hybrid_clock.cc:648] HybridClock initialized: now 1756909587928617 us; error 23 us; skew 500 ppm
Sep 03 14:26:27 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909587, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:27.930720 19741 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:27.931506 19741 webserver.cc:480] Webserver started at http://127.19.65.190:35971/ using document root <none> and password file <none>
I20250903 14:26:27.931946 19741 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:27.931983 19741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:27.932121 19741 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:27.933583 19741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "44e5799395af476594c09f32d77139e2"
format_stamp: "Formatted at 2025-09-03 14:26:27 on dist-test-slave-4wpb"
server_key: "e47b4ce9dbfa21f876d14a08e2113638"
server_key_iv: "5f4f5925bfafdc26458ad0a3197520d1"
server_key_version: "encryptionkey@0"
I20250903 14:26:27.934026 19741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "44e5799395af476594c09f32d77139e2"
format_stamp: "Formatted at 2025-09-03 14:26:27 on dist-test-slave-4wpb"
server_key: "e47b4ce9dbfa21f876d14a08e2113638"
server_key_iv: "5f4f5925bfafdc26458ad0a3197520d1"
server_key_version: "encryptionkey@0"
I20250903 14:26:27.936906 19741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:26:27.938901 19756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:27.939800 19741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:27.939894 19741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "44e5799395af476594c09f32d77139e2"
format_stamp: "Formatted at 2025-09-03 14:26:27 on dist-test-slave-4wpb"
server_key: "e47b4ce9dbfa21f876d14a08e2113638"
server_key_iv: "5f4f5925bfafdc26458ad0a3197520d1"
server_key_version: "encryptionkey@0"
I20250903 14:26:27.939967 19741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:27.959230 19741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:27.961766 19741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:27.961952 19741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:27.968546 19741 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39533
I20250903 14:26:27.968573 19808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:39533 every 8 connection(s)
I20250903 14:26:27.969514 19741 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:27.971794 19809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:27.976460 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 19741
I20250903 14:26:27.976624 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:27.976809 19809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2: Bootstrap starting.
I20250903 14:26:27.977612 19718 external_mini_cluster.cc:1468] Setting key ce5166c3f1d00bd25cfb6022c83b1c12
I20250903 14:26:27.978919 19809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:27.979676 19809 log.cc:826] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:27.981246 19809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2: No bootstrap required, opened a new log
I20250903 14:26:27.983498 19809 raft_consensus.cc:357] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } }
I20250903 14:26:27.983650 19809 raft_consensus.cc:383] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:27.983704 19809 raft_consensus.cc:738] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e5799395af476594c09f32d77139e2, State: Initialized, Role: FOLLOWER
I20250903 14:26:27.984117 19809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } }
I20250903 14:26:27.984239 19809 raft_consensus.cc:397] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:27.984311 19809 raft_consensus.cc:491] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:27.984421 19809 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:27.985132 19809 raft_consensus.cc:513] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } }
I20250903 14:26:27.985453 19809 leader_election.cc:304] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44e5799395af476594c09f32d77139e2; no voters:
I20250903 14:26:27.985760 19809 leader_election.cc:290] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:27.985886 19814 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:27.986061 19814 raft_consensus.cc:695] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [term 1 LEADER]: Becoming Leader. State: Replica: 44e5799395af476594c09f32d77139e2, State: Running, Role: LEADER
I20250903 14:26:27.986306 19814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } }
I20250903 14:26:27.986706 19809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:27.987890 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44e5799395af476594c09f32d77139e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } } }
I20250903 14:26:27.988006 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:27.988366 19823 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:27.988406 19816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44e5799395af476594c09f32d77139e2. Latest consensus state: current_term: 1 leader_uuid: "44e5799395af476594c09f32d77139e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e5799395af476594c09f32d77139e2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39533 } } }
I20250903 14:26:27.989082 19816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2 [sys.catalog]: This master's current role is: LEADER
Sep 03 14:26:27 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:27.991509 19823 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:27.995862 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:27.979456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.19.65.190:39533)
0903 14:26:27.979764 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:27.979775 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:27.980448 (+ 673us) server_negotiation.cc:408] Connection header received
0903 14:26:27.983198 (+ 2750us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:27.983211 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:27.983428 (+ 217us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:27.983665 (+ 237us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:27.985391 (+ 1726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:27.986503 (+ 1112us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:27.987306 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:27.987550 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:27.991316 (+ 3766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:27.991332 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:27.991343 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:27.991374 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:27.993350 (+ 1976us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:27.993901 (+ 551us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:27.993906 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:27.993909 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:27.993973 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:27.994257 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:27.994260 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:27.994261 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:27.994534 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:27.994739 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:27.995170 (+ 431us) server_negotiation.cc:300] Negotiation successful
0903 14:26:27.995371 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20250903 14:26:28.000319 19823 catalog_manager.cc:1349] Generated new cluster ID: f4934406769e458996b5609dc5afe5b9
I20250903 14:26:28.000375 19823 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:28.028154 19823 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:28.028963 19823 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:28.035300 19823 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 44e5799395af476594c09f32d77139e2: Generated new TSK 0
I20250903 14:26:28.035889 19823 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:28.084504 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39533
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:28.178521 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:28.178687 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:28.178743 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:28.181488 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:28.181536 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:28.181636 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:28.185308 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:39533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:28.186225 19837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:28.186946 19837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:28.192625 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:28.192633 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:28.192771 19837 server_base.cc:1047] running on GCE node
W20250903 14:26:28.192643 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:28.193176 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:28.193653 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:28.194800 19837 hybrid_clock.cc:648] HybridClock initialized: now 1756909588194784 us; error 30 us; skew 500 ppm
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:28.197376 19837 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:28.198328 19837 webserver.cc:480] Webserver started at http://127.19.65.129:45965/ using document root <none> and password file <none>
I20250903 14:26:28.198774 19837 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:28.198822 19837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:28.198987 19837 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:28.200384 19837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "33441a041b654fe782a1a81dc87f163c"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "53a97d7e5d4a323ea4fa9c6eea881482"
server_key_iv: "4d59d16134317888a1c34abe3bb6b328"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.200784 19837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "33441a041b654fe782a1a81dc87f163c"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "53a97d7e5d4a323ea4fa9c6eea881482"
server_key_iv: "4d59d16134317888a1c34abe3bb6b328"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.203495 19837 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:26:28.205384 19852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.206254 19837 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:28.206344 19837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "33441a041b654fe782a1a81dc87f163c"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "53a97d7e5d4a323ea4fa9c6eea881482"
server_key_iv: "4d59d16134317888a1c34abe3bb6b328"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.206420 19837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:28.236806 19837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:28.239223 19837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:28.239387 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:28.239873 19837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:28.240615 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:28.240660 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.240715 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:28.240747 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.248935 19837 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:33551
I20250903 14:26:28.248987 19965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:33551 every 8 connection(s)
I20250903 14:26:28.249773 19837 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:28.259025 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 19837
I20250903 14:26:28.259127 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:28.259338 19718 external_mini_cluster.cc:1468] Setting key 79835754776018148ed0b644c0a23ea8
I20250903 14:26:28.261556 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.251427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:60657 (local address 127.19.65.190:39533)
0903 14:26:28.251557 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:28.251561 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:28.252376 (+ 815us) server_negotiation.cc:408] Connection header received
0903 14:26:28.253415 (+ 1039us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:28.253419 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:28.253502 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:28.253580 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:28.254726 (+ 1146us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.255195 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.256057 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.256171 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.258190 (+ 2019us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:28.258207 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:28.258210 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:28.258228 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:28.259599 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.260108 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.260111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.260113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.260153 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.260556 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.260560 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.260562 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.260715 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:28.260809 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:28.261203 (+ 394us) server_negotiation.cc:300] Negotiation successful
0903 14:26:28.261327 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 14:26:28.261924 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.251710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39533 (local address 127.19.65.129:60657)
0903 14:26:28.252235 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:28.252276 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:28.253194 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:28.253711 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:28.253718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:28.254089 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:28.254595 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.254607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.255277 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.255279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:28.255962 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.255970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.256116 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.256626 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:28.256660 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:28.258076 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:28.259709 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.259714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.259724 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.260021 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.260226 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.260229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.260231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.260317 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.260813 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:28.260817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:28.260995 (+ 178us) client_negotiation.cc:769] Sending connection context
0903 14:26:28.261135 (+ 140us) client_negotiation.cc:240] Negotiation successful
0903 14:26:28.261306 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":100,"threads_started":1}
I20250903 14:26:28.262924 19966 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39533
I20250903 14:26:28.263144 19966 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:28.263651 19966 heartbeater.cc:507] Master 127.19.65.190:39533 requested a full tablet report, sending...
I20250903 14:26:28.265117 19773 ts_manager.cc:194] Registered new tserver with Master: 33441a041b654fe782a1a81dc87f163c (127.19.65.129:33551)
I20250903 14:26:28.266206 19773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:60657
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:28.300320 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39533
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:28.403209 19973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:28.403384 19973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:28.403463 19973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:28.406342 19973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:28.406391 19973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:28.406483 19973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:28.409884 19973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:39533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:28.410725 19973 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:28.411417 19973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:28.417294 19981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:28.417295 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:28.417449 19973 server_base.cc:1047] running on GCE node
W20250903 14:26:28.417295 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:28.417798 19973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:28.418256 19973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:28.419389 19973 hybrid_clock.cc:648] HybridClock initialized: now 1756909588419379 us; error 25 us; skew 500 ppm
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:28.421597 19973 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:28.422438 19973 webserver.cc:480] Webserver started at http://127.19.65.130:34607/ using document root <none> and password file <none>
I20250903 14:26:28.422892 19973 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:28.422940 19973 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:28.423112 19973 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:28.424651 19973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "ae8a120622cb4e06ba8c16f5efa3158a"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "ebdb97c73436d0c7c1d14622c7553925"
server_key_iv: "632c6aa1f82ab040269803c9d178e27f"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.425103 19973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "ae8a120622cb4e06ba8c16f5efa3158a"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "ebdb97c73436d0c7c1d14622c7553925"
server_key_iv: "632c6aa1f82ab040269803c9d178e27f"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.427803 19973 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20250903 14:26:28.429630 19988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.430527 19973 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:28.430627 19973 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "ae8a120622cb4e06ba8c16f5efa3158a"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "ebdb97c73436d0c7c1d14622c7553925"
server_key_iv: "632c6aa1f82ab040269803c9d178e27f"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.430701 19973 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:28.456681 19973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:28.459085 19973 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:28.459231 19973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:28.459688 19973 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:28.460433 19973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:28.460479 19973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.460536 19973 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:28.460568 19973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.468251 19973 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:37815
I20250903 14:26:28.468277 20101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:37815 every 8 connection(s)
I20250903 14:26:28.469070 19973 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:26:28.474529 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 19973
I20250903 14:26:28.474619 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:28.474826 19718 external_mini_cluster.cc:1468] Setting key c1f1bded1e1cfaedebfb6c08ed7f130f
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:28.480435 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.470547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:54517 (local address 127.19.65.190:39533)
0903 14:26:28.470705 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:28.470710 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:28.471242 (+ 532us) server_negotiation.cc:408] Connection header received
0903 14:26:28.471929 (+ 687us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:28.471931 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:28.471975 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:28.472040 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:28.473190 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.473737 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.474779 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.474986 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.477048 (+ 2062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:28.477063 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:28.477065 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:28.477087 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:28.478717 (+ 1630us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.479156 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.479175 (+ 19us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.479179 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.479233 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.479505 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.479511 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.479513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.479685 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:28.479781 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:28.480169 (+ 388us) server_negotiation.cc:300] Negotiation successful
0903 14:26:28.480293 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250903 14:26:28.481119 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.470763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39533 (local address 127.19.65.130:54517)
0903 14:26:28.471116 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:28.471147 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:28.471768 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:28.472207 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:28.472214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:28.472550 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:28.473071 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.473083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.473870 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.473874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:28.474642 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.474654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.474857 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.475411 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:28.475444 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:28.476900 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:28.478834 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.478838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.478845 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.479061 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.479340 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.479343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.479345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.479420 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.479774 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:28.479778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:28.479955 (+ 177us) client_negotiation.cc:769] Sending connection context
0903 14:26:28.480114 (+ 159us) client_negotiation.cc:240] Negotiation successful
0903 14:26:28.480331 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20250903 14:26:28.482110 20102 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39533
I20250903 14:26:28.482304 20102 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:28.482734 20102 heartbeater.cc:507] Master 127.19.65.190:39533 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20250903 14:26:28.483639 19773 ts_manager.cc:194] Registered new tserver with Master: ae8a120622cb4e06ba8c16f5efa3158a (127.19.65.130:37815)
I20250903 14:26:28.484210 19773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:54517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:28.520028 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39533
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:28.614473 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:28.614655 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:28.614732 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:28.617599 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:28.617652 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:28.617774 20109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:28.621420 20109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:39533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:28.622349 20109 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:28.623104 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:28.629053 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:28.629088 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:28.629099 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:28.629243 20109 server_base.cc:1047] running on GCE node
I20250903 14:26:28.629602 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:28.630118 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:28.631255 20109 hybrid_clock.cc:648] HybridClock initialized: now 1756909588631244 us; error 31 us; skew 500 ppm
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:28.633814 20109 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:28.634814 20109 webserver.cc:480] Webserver started at http://127.19.65.131:45863/ using document root <none> and password file <none>
I20250903 14:26:28.635305 20109 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:28.635353 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:28.635510 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:28.637084 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "aa76faa0d7364e29804c621d37a7ba74"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "bd213349a9bfa156bdd9e65faed40294"
server_key_iv: "1b1c7366ed2f9e1ba8e6c06b779988b1"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.637565 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "aa76faa0d7364e29804c621d37a7ba74"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "bd213349a9bfa156bdd9e65faed40294"
server_key_iv: "1b1c7366ed2f9e1ba8e6c06b779988b1"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.640486 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 14:26:28.642343 20124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.643277 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:28.643369 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "aa76faa0d7364e29804c621d37a7ba74"
format_stamp: "Formatted at 2025-09-03 14:26:28 on dist-test-slave-4wpb"
server_key: "bd213349a9bfa156bdd9e65faed40294"
server_key_iv: "1b1c7366ed2f9e1ba8e6c06b779988b1"
server_key_version: "encryptionkey@0"
I20250903 14:26:28.643448 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:28.663874 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:28.666659 20109 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:28.666796 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:28.667232 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:28.667977 20109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:28.668021 20109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.668058 20109 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:28.668078 20109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:28.676149 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:42685
I20250903 14:26:28.676200 20237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:42685 every 8 connection(s)
I20250903 14:26:28.677084 20109 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:28.684346 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20109
I20250903 14:26:28.684435 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:28.684644 19718 external_mini_cluster.cc:1468] Setting key 970b196383958b7c97f3cc7584fe28be
I20250903 14:26:28.688098 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.678718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59937 (local address 127.19.65.190:39533)
0903 14:26:28.678850 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:28.678854 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:28.679428 (+ 574us) server_negotiation.cc:408] Connection header received
0903 14:26:28.680185 (+ 757us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:28.680189 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:28.680246 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:28.680317 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:28.681509 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.681998 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.682897 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.683059 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.685045 (+ 1986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:28.685069 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:28.685072 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:28.685095 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:28.686566 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.687024 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.687027 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.687028 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.687067 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.687259 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.687261 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.687262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.687404 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:28.687470 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:28.687884 (+ 414us) server_negotiation.cc:300] Negotiation successful
0903 14:26:28.687971 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 14:26:28.688648 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.678948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39533 (local address 127.19.65.131:59937)
0903 14:26:28.679307 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:28.679339 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:28.680000 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:28.680486 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:28.680493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:28.680840 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:28.681348 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.681359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.682110 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.682112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:28.682787 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:28.682794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.682975 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.683488 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:28.683508 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:28.684908 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:28.686672 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.686677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.686686 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.686954 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.687133 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:28.687135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:28.687136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:28.687200 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:28.687480 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:28.687485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:28.687664 (+ 179us) client_negotiation.cc:769] Sending connection context
0903 14:26:28.687849 (+ 185us) client_negotiation.cc:240] Negotiation successful
0903 14:26:28.688074 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20250903 14:26:28.689517 20238 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39533
I20250903 14:26:28.689736 20238 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:28.690165 20238 heartbeater.cc:507] Master 127.19.65.190:39533 requested a full tablet report, sending...
I20250903 14:26:28.690990 19773 ts_manager.cc:194] Registered new tserver with Master: aa76faa0d7364e29804c621d37a7ba74 (127.19.65.131:42685)
I20250903 14:26:28.691447 19773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:59937
I20250903 14:26:28.698485 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:28.719866 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.712065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.19.65.129:33551)
0903 14:26:28.712316 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:28.712321 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:28.712344 (+ 23us) server_negotiation.cc:408] Connection header received
0903 14:26:28.712416 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:28.712420 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:28.712578 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:28.712697 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:28.713350 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.714131 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.714730 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.714921 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.716756 (+ 1835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:28.716797 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:28.716808 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:28.716832 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:28.718510 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.718854 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.718859 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.718861 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.718908 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.719069 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.719070 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.719072 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.719237 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:28.719381 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:28.719507 (+ 126us) server_negotiation.cc:300] Negotiation successful
0903 14:26:28.719636 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
W20250903 14:26:28.720753 19880 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:28.742359 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:28.735519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.19.65.129:33551)
0903 14:26:28.735682 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:28.735689 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:28.735742 (+ 53us) server_negotiation.cc:408] Connection header received
0903 14:26:28.735876 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:28.735879 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:28.735928 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:28.735992 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:28.736729 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.737216 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:28.737877 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:28.738075 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:28.739881 (+ 1806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:28.739897 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:28.739899 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:28.739921 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:28.741071 (+ 1150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.741423 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.741427 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.741428 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.741466 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:28.741720 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:28.741722 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:28.741723 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:28.741855 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:28.741963 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:28.742101 (+ 138us) server_negotiation.cc:300] Negotiation successful
0903 14:26:28.742186 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20250903 14:26:28.743232 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 19837
I20250903 14:26:28.748725 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 19973
I20250903 14:26:28.755625 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20109
I20250903 14:26:28.760633 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 19741
2025-09-03T14:26:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2928 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[20252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[20252](info): set up 2 sockets
Sep 03 14:26:28 dist-test-slave-4wpb krb5kdc[20252](info): commencing operation
krb5kdc: starting...
W20250903 14:26:30.589295 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.807s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:30 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:30.710039 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45805
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:30.801175 20268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:30.801352 20268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:30.801430 20268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:30.804136 20268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:30.804191 20268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:30.804232 20268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:30.804265 20268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:30.804303 20268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:30.807835 20268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:30.808704 20268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:30.809404 20268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:30.814071 20274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:30.814154 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:30.814155 20273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:30.814229 20268 server_base.cc:1047] running on GCE node
I20250903 14:26:30.814767 20268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:30.815510 20268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:30.816655 20268 hybrid_clock.cc:648] HybridClock initialized: now 1756909590816635 us; error 32 us; skew 500 ppm
Sep 03 14:26:30 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909590, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:30.818843 20268 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:30.819691 20268 webserver.cc:480] Webserver started at http://127.19.65.190:45867/ using document root <none> and password file <none>
I20250903 14:26:30.820122 20268 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:30.820163 20268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:30.820291 20268 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:30.821796 20268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "0bf0f36478f74b15a50596e26d9e243c"
format_stamp: "Formatted at 2025-09-03 14:26:30 on dist-test-slave-4wpb"
server_key: "d5fcb965ac70318d3fbfe40ca7d89565"
server_key_iv: "353cfac3acb42aeddc20822b7374df24"
server_key_version: "encryptionkey@0"
I20250903 14:26:30.822181 20268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "0bf0f36478f74b15a50596e26d9e243c"
format_stamp: "Formatted at 2025-09-03 14:26:30 on dist-test-slave-4wpb"
server_key: "d5fcb965ac70318d3fbfe40ca7d89565"
server_key_iv: "353cfac3acb42aeddc20822b7374df24"
server_key_version: "encryptionkey@0"
I20250903 14:26:30.825065 20268 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:26:30.826790 20283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:30.827584 20268 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 14:26:30.827666 20268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "0bf0f36478f74b15a50596e26d9e243c"
format_stamp: "Formatted at 2025-09-03 14:26:30 on dist-test-slave-4wpb"
server_key: "d5fcb965ac70318d3fbfe40ca7d89565"
server_key_iv: "353cfac3acb42aeddc20822b7374df24"
server_key_version: "encryptionkey@0"
I20250903 14:26:30.827728 20268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:30.837913 20268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:30.846006 20268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:30.846134 20268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:30.852833 20335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:45805 every 8 connection(s)
I20250903 14:26:30.852833 20268 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45805
I20250903 14:26:30.853881 20268 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:30.856209 20336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:30.861025 20336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: Bootstrap starting.
I20250903 14:26:30.863109 20336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:30.863776 20336 log.cc:826] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:30.864030 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20268
I20250903 14:26:30.864140 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:30.864405 19718 external_mini_cluster.cc:1468] Setting key ffd6934f865a1ba71595ce268df2bf4f
I20250903 14:26:30.865712 20336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: No bootstrap required, opened a new log
I20250903 14:26:30.868367 20336 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } }
I20250903 14:26:30.868559 20336 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:30.868640 20336 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf0f36478f74b15a50596e26d9e243c, State: Initialized, Role: FOLLOWER
I20250903 14:26:30.869056 20336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } }
I20250903 14:26:30.869168 20336 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:30.869225 20336 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:30.869374 20336 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:30.870317 20336 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } }
I20250903 14:26:30.870658 20336 leader_election.cc:304] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bf0f36478f74b15a50596e26d9e243c; no voters:
I20250903 14:26:30.870910 20336 leader_election.cc:290] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:30.870996 20341 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:30.871172 20341 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [term 1 LEADER]: Becoming Leader. State: Replica: 0bf0f36478f74b15a50596e26d9e243c, State: Running, Role: LEADER
Sep 03 14:26:30 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:30.871488 20341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } }
I20250903 14:26:30.871970 20336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:30.873080 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf0f36478f74b15a50596e26d9e243c. Latest consensus state: current_term: 1 leader_uuid: "0bf0f36478f74b15a50596e26d9e243c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } } }
I20250903 14:26:30.873195 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:30.873309 20343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf0f36478f74b15a50596e26d9e243c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0f36478f74b15a50596e26d9e243c" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45805 } } }
I20250903 14:26:30.874004 20343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c [sys.catalog]: This master's current role is: LEADER
W20250903 14:26:30.876869 20356 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:26:30.876969 20356 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:26:30.877101 20355 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:30.877650 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:30.865373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.19.65.190:45805)
0903 14:26:30.865786 (+ 413us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:30.865796 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:30.865829 (+ 33us) server_negotiation.cc:408] Connection header received
0903 14:26:30.866409 (+ 580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:30.866431 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:30.866738 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:30.867034 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:30.867688 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:30.868770 (+ 1082us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:30.869540 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:30.869847 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:30.872010 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:30.872030 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:30.872043 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:30.872086 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:30.875092 (+ 3006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:30.875439 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:30.875443 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:30.875447 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:30.875506 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:30.875757 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:30.875760 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:30.875762 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:30.876137 (+ 375us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:30.876309 (+ 172us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:30.876635 (+ 326us) server_negotiation.cc:300] Negotiation successful
0903 14:26:30.876870 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20250903 14:26:30.877980 20355 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:30.882678 20355 catalog_manager.cc:1349] Generated new cluster ID: f6449790fd1745d988f62c2d9d8acf6f
I20250903 14:26:30.882735 20355 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:30.902938 20355 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:30.903764 20355 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:30.913338 20355 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0bf0f36478f74b15a50596e26d9e243c: Generated new TSK 0
I20250903 14:26:30.914023 20355 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:30.962057 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45805
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:31.059305 20364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:31.059506 20364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:31.059583 20364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:31.062779 20364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:31.062832 20364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:31.062938 20364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:31.066887 20364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:31.067849 20364 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:31.068694 20364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:31.074566 20369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:31.074574 20372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:31.074770 20364 server_base.cc:1047] running on GCE node
W20250903 14:26:31.074563 20370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:31.075229 20364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:31.075693 20364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:31.076836 20364 hybrid_clock.cc:648] HybridClock initialized: now 1756909591076818 us; error 33 us; skew 500 ppm
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:31.079303 20364 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:31.080222 20364 webserver.cc:480] Webserver started at http://127.19.65.129:42397/ using document root <none> and password file <none>
I20250903 14:26:31.080727 20364 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:31.080780 20364 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:31.080950 20364 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:31.082636 20364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "015f2dc9da0c43ad876acaecd31e81b9"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "b03648b04b0b20ccb2f9a9e4d67ba036"
server_key_iv: "2d7b7710cac8bcd4f8654e344fd99578"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.083077 20364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "015f2dc9da0c43ad876acaecd31e81b9"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "b03648b04b0b20ccb2f9a9e4d67ba036"
server_key_iv: "2d7b7710cac8bcd4f8654e344fd99578"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.086306 20364 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 14:26:31.088280 20379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.089160 20364 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:31.089267 20364 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "015f2dc9da0c43ad876acaecd31e81b9"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "b03648b04b0b20ccb2f9a9e4d67ba036"
server_key_iv: "2d7b7710cac8bcd4f8654e344fd99578"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.089352 20364 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:31.098470 20364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:31.101137 20364 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:31.101323 20364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:31.101917 20364 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:31.102754 20364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:31.102800 20364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.102859 20364 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:31.102888 20364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.111696 20364 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:38323
I20250903 14:26:31.111728 20492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:38323 every 8 connection(s)
I20250903 14:26:31.112603 20364 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:26:31.117450 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20364
I20250903 14:26:31.117605 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:31.117831 19718 external_mini_cluster.cc:1468] Setting key 9a1c629a61210ae698d383cefc518a1c
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:31.125504 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.114302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37027 (local address 127.19.65.190:45805)
0903 14:26:31.114439 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.114443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.115048 (+ 605us) server_negotiation.cc:408] Connection header received
0903 14:26:31.115819 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.115822 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.115869 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.115942 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.117197 (+ 1255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.117838 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.119227 (+ 1389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.119402 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.121677 (+ 2275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.121698 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.121705 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.121737 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.123600 (+ 1863us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.124052 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.124057 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.124061 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.124117 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.124365 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.124371 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.124375 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.124604 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.124713 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.125141 (+ 428us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.125281 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 14:26:31.126029 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.114522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45805 (local address 127.19.65.129:37027)
0903 14:26:31.114906 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.114939 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.115647 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.116076 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.116084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.116486 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:31.117036 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.117048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.117999 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.118003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.119117 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.119126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.119328 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.119938 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:31.119961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:31.121523 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:31.123720 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.123724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.123733 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.123950 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.124204 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.124206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.124208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.124278 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.124697 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:31.124701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:31.124906 (+ 205us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.125070 (+ 164us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.125271 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:26:31.127223 20493 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45805
I20250903 14:26:31.127467 20493 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:31.127985 20493 heartbeater.cc:507] Master 127.19.65.190:45805 requested a full tablet report, sending...
I20250903 14:26:31.129518 20300 ts_manager.cc:194] Registered new tserver with Master: 015f2dc9da0c43ad876acaecd31e81b9 (127.19.65.129:38323)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
I20250903 14:26:31.131124 20300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:37027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:31.165140 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45805
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:31.271497 20500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:31.271715 20500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:31.271772 20500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:31.275153 20500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:31.275216 20500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:31.275305 20500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:31.279433 20500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:31.280520 20500 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:31.281322 20500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:31.287765 20506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:31.287806 20508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:31.287765 20505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:31.287814 20500 server_base.cc:1047] running on GCE node
I20250903 14:26:31.288373 20500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:31.288916 20500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:31.290081 20500 hybrid_clock.cc:648] HybridClock initialized: now 1756909591290060 us; error 33 us; skew 500 ppm
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:31.292914 20500 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:31.294014 20500 webserver.cc:480] Webserver started at http://127.19.65.130:40383/ using document root <none> and password file <none>
I20250903 14:26:31.294580 20500 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:31.294636 20500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:31.294818 20500 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:31.296602 20500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "cd90072dd340481b84b819fe4523e927"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "ab15f7490ebd1b984e71c0e99833c450"
server_key_iv: "bd08082ec57d27be702c82e5a392171b"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.297066 20500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "cd90072dd340481b84b819fe4523e927"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "ab15f7490ebd1b984e71c0e99833c450"
server_key_iv: "bd08082ec57d27be702c82e5a392171b"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.300727 20500 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250903 14:26:31.302949 20515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.303925 20500 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 14:26:31.304034 20500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "cd90072dd340481b84b819fe4523e927"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "ab15f7490ebd1b984e71c0e99833c450"
server_key_iv: "bd08082ec57d27be702c82e5a392171b"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.304134 20500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:31.337409 20500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:31.340448 20500 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:31.340641 20500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:31.341184 20500 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:31.342082 20500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:31.342132 20500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.342199 20500 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:31.342227 20500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.351375 20500 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41875
I20250903 14:26:31.351402 20628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:41875 every 8 connection(s)
I20250903 14:26:31.352349 20500 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:31.361120 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20500
I20250903 14:26:31.361246 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:31.361552 19718 external_mini_cluster.cc:1468] Setting key 813fdd63249731b2645beac3b219ee7a
I20250903 14:26:31.365687 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.354158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:45731 (local address 127.19.65.190:45805)
0903 14:26:31.354330 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.354333 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.355008 (+ 675us) server_negotiation.cc:408] Connection header received
0903 14:26:31.355895 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.355899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.355961 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.356039 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.357334 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.357912 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.359034 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.359209 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.361571 (+ 2362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.361598 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.361603 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.361640 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.363349 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.364052 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.364055 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.364056 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.364100 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.364568 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.364571 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.364572 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.364744 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.364830 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.365358 (+ 528us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.365515 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20250903 14:26:31.366343 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.354419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45805 (local address 127.19.65.130:45731)
0903 14:26:31.354856 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.354893 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.355685 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.356209 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.356216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.356640 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:31.357175 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.357188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.358034 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.358037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.358913 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.358921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.359127 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.359795 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:31.359819 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:31.361341 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:31.363545 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.363552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.363566 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.363943 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.364190 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.364197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.364202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.364439 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.364854 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:31.364861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:31.365095 (+ 234us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.365308 (+ 213us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.365635 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20250903 14:26:31.367615 20629 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45805
I20250903 14:26:31.367849 20629 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:31.368366 20629 heartbeater.cc:507] Master 127.19.65.190:45805 requested a full tablet report, sending...
I20250903 14:26:31.369472 20300 ts_manager.cc:194] Registered new tserver with Master: cd90072dd340481b84b819fe4523e927 (127.19.65.130:41875)
I20250903 14:26:31.370276 20300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:45731
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:31.408200 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45805
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:31.508452 20636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:31.508648 20636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:31.508719 20636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:31.511561 20636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:31.511615 20636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:31.511686 20636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:31.515467 20636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:31.516399 20636 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:31.517135 20636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:31.523072 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:31.523072 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:31.523067 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:31.523293 20636 server_base.cc:1047] running on GCE node
I20250903 14:26:31.523684 20636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:31.524173 20636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:31.525316 20636 hybrid_clock.cc:648] HybridClock initialized: now 1756909591525302 us; error 28 us; skew 500 ppm
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:31.527824 20636 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:31.528828 20636 webserver.cc:480] Webserver started at http://127.19.65.131:40111/ using document root <none> and password file <none>
I20250903 14:26:31.529358 20636 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:31.529409 20636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:31.529615 20636 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:31.531255 20636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "476ec43ad58a48eca2e8e03a9f70e6ec"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "d079c7ae2e075781134e512a4dc3b441"
server_key_iv: "6b1cf1d563648a9a922bbced0051e9c4"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.531675 20636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "476ec43ad58a48eca2e8e03a9f70e6ec"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "d079c7ae2e075781134e512a4dc3b441"
server_key_iv: "6b1cf1d563648a9a922bbced0051e9c4"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.534795 20636 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:26:31.536703 20651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.537617 20636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:26:31.537707 20636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "476ec43ad58a48eca2e8e03a9f70e6ec"
format_stamp: "Formatted at 2025-09-03 14:26:31 on dist-test-slave-4wpb"
server_key: "d079c7ae2e075781134e512a4dc3b441"
server_key_iv: "6b1cf1d563648a9a922bbced0051e9c4"
server_key_version: "encryptionkey@0"
I20250903 14:26:31.537782 20636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:31.561434 20636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:31.564251 20636 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:31.564415 20636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:31.564924 20636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:31.565716 20636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:31.565761 20636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.565830 20636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:31.565855 20636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:31.573455 20636 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:37869
I20250903 14:26:31.573541 20764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:37869 every 8 connection(s)
I20250903 14:26:31.574432 20636 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:31.583324 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20636
I20250903 14:26:31.583407 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:31.583642 19718 external_mini_cluster.cc:1468] Setting key fa53ed84042d7dab39647b0067e99e6b
I20250903 14:26:31.585458 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.576066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:36059 (local address 127.19.65.190:45805)
0903 14:26:31.576199 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.576203 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.576860 (+ 657us) server_negotiation.cc:408] Connection header received
0903 14:26:31.577642 (+ 782us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.577645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.577711 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.577799 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.578882 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.579332 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.580293 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.580401 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.582232 (+ 1831us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.582265 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.582267 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.582289 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.583613 (+ 1324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.584122 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.584126 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.584128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.584177 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.584443 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.584446 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.584448 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.584636 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.584750 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.585260 (+ 510us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.585351 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 14:26:31.586148 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.576336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45805 (local address 127.19.65.131:36059)
0903 14:26:31.576734 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.576767 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.577405 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.577926 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.577933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.578272 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:31.578768 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.578780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.579411 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.579414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.580206 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.580212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.580349 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.580830 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:31.580848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:31.582108 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:31.583728 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.583733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.583743 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.584018 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.584273 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:31.584278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:31.584280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:31.584355 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:31.584770 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:31.584779 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:31.585001 (+ 222us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.585208 (+ 207us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.585455 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20250903 14:26:31.587059 20765 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45805
I20250903 14:26:31.587275 20765 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:31.587728 20765 heartbeater.cc:507] Master 127.19.65.190:45805 requested a full tablet report, sending...
I20250903 14:26:31.588567 20300 ts_manager.cc:194] Registered new tserver with Master: 476ec43ad58a48eca2e8e03a9f70e6ec (127.19.65.131:37869)
I20250903 14:26:31.589027 20300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:36059
I20250903 14:26:31.597142 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:26:31.605643 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.599683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.19.65.190:45805)
0903 14:26:31.599814 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.599817 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.599930 (+ 113us) server_negotiation.cc:408] Connection header received
0903 14:26:31.600073 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.600077 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.600118 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.600178 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.600914 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.601343 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.601979 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.602170 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.603031 (+ 861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.603046 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.603049 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.603069 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.604261 (+ 1192us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.604638 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.604642 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.604643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.604683 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.604880 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.604883 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.604885 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.605024 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.605135 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.605378 (+ 243us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.605474 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:26:31.608976 20300 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:31.611114 20300 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:31.621858 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.618332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:37869 (local address 127.0.0.1:44060)
0903 14:26:31.618594 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.618607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.618672 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.619338 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.619344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.619366 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:31.619637 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.619644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.620779 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.620784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.621410 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.621418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.621548 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.621591 (+ 43us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.621667 (+ 76us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.621739 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20250903 14:26:31.622179 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.618577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38323 (local address 127.0.0.1:49772)
0903 14:26:31.618909 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.618919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.618978 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.619626 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.619629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.619646 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:31.619840 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.619846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.620947 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.620950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.621857 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.621867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.621964 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.621979 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.622023 (+ 44us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.622071 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":81,"threads_started":1}
I20250903 14:26:31.622427 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.618561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.19.65.131:37869)
0903 14:26:31.618886 (+ 325us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.618892 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.618915 (+ 23us) server_negotiation.cc:408] Connection header received
0903 14:26:31.618997 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.619002 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.619176 (+ 174us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.619328 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:31.619729 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.620676 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.621564 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.622027 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.622123 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.622177 (+ 54us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.622292 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
I20250903 14:26:31.622886 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.617965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41875 (local address 127.0.0.1:59490)
0903 14:26:31.618409 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.618449 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.618592 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.619982 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.619985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.620002 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:31.620213 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.620219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.621522 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.621525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.622550 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.622560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.622668 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.622683 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.622731 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.622781 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":72,"threads_started":1}
I20250903 14:26:31.623528 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.618135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.19.65.130:41875)
0903 14:26:31.619605 (+ 1470us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.619610 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.619629 (+ 19us) server_negotiation.cc:408] Connection header received
0903 14:26:31.619705 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.619710 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.619853 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.619982 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:31.620331 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.621387 (+ 1056us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.622662 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.623179 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.623301 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.623351 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.623410 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":131,"threads_started":1}
I20250903 14:26:31.623719 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.618912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.19.65.129:38323)
0903 14:26:31.619221 (+ 309us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.619226 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.619245 (+ 19us) server_negotiation.cc:408] Connection header received
0903 14:26:31.619326 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.619331 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.619475 (+ 144us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.619950 (+ 475us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:31.620079 (+ 129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.620840 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.622874 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.623336 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.623424 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.623514 (+ 90us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.623594 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":117,"threads_started":1}
I20250903 14:26:31.624820 20699 tablet_service.cc:1468] Processing CreateTablet for tablet 9700b141451141ae967304fa76191af7 (DEFAULT_TABLE table=test-table [id=14dffb9b4ee84dfd99a0e75d4d51dec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:31.625706 20699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700b141451141ae967304fa76191af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:31.625627 20562 tablet_service.cc:1468] Processing CreateTablet for tablet 9700b141451141ae967304fa76191af7 (DEFAULT_TABLE table=test-table [id=14dffb9b4ee84dfd99a0e75d4d51dec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:31.625627 20427 tablet_service.cc:1468] Processing CreateTablet for tablet 9700b141451141ae967304fa76191af7 (DEFAULT_TABLE table=test-table [id=14dffb9b4ee84dfd99a0e75d4d51dec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:31.626439 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700b141451141ae967304fa76191af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:31.626439 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700b141451141ae967304fa76191af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:31.630656 20782 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: Bootstrap starting.
I20250903 14:26:31.631848 20783 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: Bootstrap starting.
I20250903 14:26:31.632411 20784 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: Bootstrap starting.
I20250903 14:26:31.632783 20782 tablet_bootstrap.cc:654] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:31.633608 20782 log.cc:826] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:31.633800 20783 tablet_bootstrap.cc:654] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:31.634007 20784 tablet_bootstrap.cc:654] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:31.634627 20783 log.cc:826] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:31.634840 20784 log.cc:826] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:31.635547 20782 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: No bootstrap required, opened a new log
I20250903 14:26:31.635718 20782 ts_tablet_manager.cc:1397] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:26:31.635951 20783 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: No bootstrap required, opened a new log
I20250903 14:26:31.636107 20783 ts_tablet_manager.cc:1397] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:26:31.636988 20784 tablet_bootstrap.cc:492] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: No bootstrap required, opened a new log
I20250903 14:26:31.637148 20784 ts_tablet_manager.cc:1397] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:26:31.638302 20782 raft_consensus.cc:357] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.638468 20782 raft_consensus.cc:383] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:31.638540 20782 raft_consensus.cc:738] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476ec43ad58a48eca2e8e03a9f70e6ec, State: Initialized, Role: FOLLOWER
I20250903 14:26:31.638924 20782 consensus_queue.cc:260] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.638942 20783 raft_consensus.cc:357] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.639168 20783 raft_consensus.cc:383] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:31.639276 20783 raft_consensus.cc:738] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015f2dc9da0c43ad876acaecd31e81b9, State: Initialized, Role: FOLLOWER
I20250903 14:26:31.639540 20765 heartbeater.cc:499] Master 127.19.65.190:45805 was elected leader, sending a full tablet report...
I20250903 14:26:31.639739 20782 ts_tablet_manager.cc:1428] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250903 14:26:31.639716 20783 consensus_queue.cc:260] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.640278 20493 heartbeater.cc:499] Master 127.19.65.190:45805 was elected leader, sending a full tablet report...
I20250903 14:26:31.640452 20783 ts_tablet_manager.cc:1428] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:26:31.641157 20784 raft_consensus.cc:357] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.641355 20784 raft_consensus.cc:383] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:31.641420 20784 raft_consensus.cc:738] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd90072dd340481b84b819fe4523e927, State: Initialized, Role: FOLLOWER
I20250903 14:26:31.641808 20784 consensus_queue.cc:260] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.642310 20629 heartbeater.cc:499] Master 127.19.65.190:45805 was elected leader, sending a full tablet report...
I20250903 14:26:31.642509 20784 ts_tablet_manager.cc:1428] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 14:26:31.749334 20788 raft_consensus.cc:491] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:31.749531 20788 raft_consensus.cc:513] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.750542 20788 leader_election.cc:290] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd90072dd340481b84b819fe4523e927 (127.19.65.130:41875), 015f2dc9da0c43ad876acaecd31e81b9 (127.19.65.129:38323)
I20250903 14:26:31.753392 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.750717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41875 (local address 127.19.65.131:47685)
0903 14:26:31.750843 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.750855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.750944 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.751240 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.751243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.751265 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:31.751500 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.751508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.752303 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.752305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.753035 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.753043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.753143 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.753161 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.753202 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.753260 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20250903 14:26:31.753633 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.750954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38323 (local address 127.19.65.131:54165)
0903 14:26:31.751244 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:31.751255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:31.751335 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:31.751534 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:31.751539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:31.751560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:31.751757 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.751762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.752543 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.752548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:31.753256 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:31.753263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.753349 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.753361 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 14:26:31.753402 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:26:31.753457 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20250903 14:26:31.753825 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.750839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:47685 (local address 127.19.65.130:41875)
0903 14:26:31.750969 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.750975 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.750991 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:26:31.751046 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.751048 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.751084 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.751151 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:31.751612 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.752204 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.753139 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.753565 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.753605 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.753653 (+ 48us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.753704 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 14:26:31.754086 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.751056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:54165 (local address 127.19.65.129:38323)
0903 14:26:31.751187 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.751191 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.751316 (+ 125us) server_negotiation.cc:408] Connection header received
0903 14:26:31.751411 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.751413 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.751452 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.751506 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:31.751850 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.752443 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.753356 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.753836 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.753873 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.753923 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.753981 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:26:31.754382 20583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9700b141451141ae967304fa76191af7" candidate_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd90072dd340481b84b819fe4523e927" is_pre_election: true
I20250903 14:26:31.754585 20447 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9700b141451141ae967304fa76191af7" candidate_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015f2dc9da0c43ad876acaecd31e81b9" is_pre_election: true
I20250903 14:26:31.754638 20583 raft_consensus.cc:2466] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476ec43ad58a48eca2e8e03a9f70e6ec in term 0.
I20250903 14:26:31.754827 20447 raft_consensus.cc:2466] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476ec43ad58a48eca2e8e03a9f70e6ec in term 0.
I20250903 14:26:31.755075 20653 leader_election.cc:304] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476ec43ad58a48eca2e8e03a9f70e6ec, cd90072dd340481b84b819fe4523e927; no voters:
I20250903 14:26:31.755283 20788 raft_consensus.cc:2802] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:31.755344 20788 raft_consensus.cc:491] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:31.755371 20788 raft_consensus.cc:3058] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:31.756165 20788 raft_consensus.cc:513] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.756454 20788 leader_election.cc:290] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [CANDIDATE]: Term 1 election: Requested vote from peers cd90072dd340481b84b819fe4523e927 (127.19.65.130:41875), 015f2dc9da0c43ad876acaecd31e81b9 (127.19.65.129:38323)
I20250903 14:26:31.756795 20583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9700b141451141ae967304fa76191af7" candidate_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd90072dd340481b84b819fe4523e927"
I20250903 14:26:31.756816 20447 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9700b141451141ae967304fa76191af7" candidate_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015f2dc9da0c43ad876acaecd31e81b9"
I20250903 14:26:31.756897 20583 raft_consensus.cc:3058] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:31.756897 20447 raft_consensus.cc:3058] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:31.758060 20583 raft_consensus.cc:2466] T 9700b141451141ae967304fa76191af7 P cd90072dd340481b84b819fe4523e927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476ec43ad58a48eca2e8e03a9f70e6ec in term 1.
I20250903 14:26:31.758312 20653 leader_election.cc:304] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476ec43ad58a48eca2e8e03a9f70e6ec, cd90072dd340481b84b819fe4523e927; no voters:
I20250903 14:26:31.758415 20447 raft_consensus.cc:2466] T 9700b141451141ae967304fa76191af7 P 015f2dc9da0c43ad876acaecd31e81b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476ec43ad58a48eca2e8e03a9f70e6ec in term 1.
I20250903 14:26:31.758472 20788 raft_consensus.cc:2802] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:31.758625 20788 raft_consensus.cc:695] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [term 1 LEADER]: Becoming Leader. State: Replica: 476ec43ad58a48eca2e8e03a9f70e6ec, State: Running, Role: LEADER
I20250903 14:26:31.758901 20788 consensus_queue.cc:237] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } }
I20250903 14:26:31.761675 20298 catalog_manager.cc:5582] T 9700b141451141ae967304fa76191af7 P 476ec43ad58a48eca2e8e03a9f70e6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 476ec43ad58a48eca2e8e03a9f70e6ec (127.19.65.131). New cstate: current_term: 1 leader_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd90072dd340481b84b819fe4523e927" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015f2dc9da0c43ad876acaecd31e81b9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476ec43ad58a48eca2e8e03a9f70e6ec" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37869 } health_report { overall_health: HEALTHY } } }
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:31.771752 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.765274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.19.65.129:38323)
0903 14:26:31.765374 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.765378 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.765522 (+ 144us) server_negotiation.cc:408] Connection header received
0903 14:26:31.765642 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.765645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.765678 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.765736 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.766462 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.766850 (+ 388us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.767387 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.767544 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.769436 (+ 1892us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.769451 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.769462 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.769510 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.770763 (+ 1253us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.771067 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.771071 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.771072 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.771110 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.771245 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.771247 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.771248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.771370 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.771492 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.771587 (+ 95us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.771663 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:31.791940 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.785964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.19.65.129:38323)
0903 14:26:31.786069 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.786072 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.786179 (+ 107us) server_negotiation.cc:408] Connection header received
0903 14:26:31.786263 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.786265 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.786297 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.786353 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.786929 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.787303 (+ 374us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.787786 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.787881 (+ 95us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.789769 (+ 1888us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.789790 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.789794 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.789824 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.790965 (+ 1141us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.791273 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.791275 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.791276 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.791322 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.791468 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.791469 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.791470 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.791557 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.791640 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.791776 (+ 136us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.791845 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:31.810563 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:31.804544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.19.65.129:38323)
0903 14:26:31.804667 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:31.804671 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:31.804784 (+ 113us) server_negotiation.cc:408] Connection header received
0903 14:26:31.804899 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:31.804902 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:31.804935 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:31.804995 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:31.805715 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.806085 (+ 370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:31.806604 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:31.806775 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:31.808431 (+ 1656us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:31.808444 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:31.808446 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:31.808464 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:31.809667 (+ 1203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.809939 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.809942 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.809943 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.809972 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:31.810102 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:31.810103 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:31.810104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:31.810187 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:31.810292 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:31.810407 (+ 115us) server_negotiation.cc:300] Negotiation successful
0903 14:26:31.810471 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20250903 14:26:31.811655 20407 tablet_service.cc:2912] Rejecting scan request for tablet 9700b141451141ae967304fa76191af7: Uninitialized: clean time has not yet been initialized
I20250903 14:26:31.813218 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20364
I20250903 14:26:31.818979 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20500
I20250903 14:26:31.824004 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20636
I20250903 14:26:31.828838 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20268
2025-09-03T14:26:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3068 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20806](info): set up 2 sockets
Sep 03 14:26:31 dist-test-slave-4wpb krb5kdc[20806](info): commencing operation
krb5kdc: starting...
W20250903 14:26:33.625239 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.775s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:33 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:33.739418 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42439
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:33.827883 20822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:33.828052 20822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:33.828130 20822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:33.830770 20822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:33.830816 20822 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 14:26:33.830842 20822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:33.830876 20822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:33.830911 20822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:33.830941 20822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:33.834571 20822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:33.835362 20822 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:33.836041 20822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:33.840507 20830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:33.840462 20827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:33.840572 20828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:33.840827 20822 server_base.cc:1047] running on GCE node
I20250903 14:26:33.841173 20822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:33.841899 20822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:33.843032 20822 hybrid_clock.cc:648] HybridClock initialized: now 1756909593843025 us; error 26 us; skew 500 ppm
Sep 03 14:26:33 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909593, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:33.845364 20822 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:33.846221 20822 webserver.cc:480] Webserver started at http://127.19.65.190:40329/ using document root <none> and password file <none>
I20250903 14:26:33.846653 20822 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:33.846701 20822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:33.846853 20822 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:33.848263 20822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "782c68d2503343918810c2bcbb6885b6"
format_stamp: "Formatted at 2025-09-03 14:26:33 on dist-test-slave-4wpb"
server_key: "ef60af3a724ae37ef1628051c602844b"
server_key_iv: "74fb7244bd82de0eca016b4df8444099"
server_key_version: "encryptionkey@0"
I20250903 14:26:33.848670 20822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "782c68d2503343918810c2bcbb6885b6"
format_stamp: "Formatted at 2025-09-03 14:26:33 on dist-test-slave-4wpb"
server_key: "ef60af3a724ae37ef1628051c602844b"
server_key_iv: "74fb7244bd82de0eca016b4df8444099"
server_key_version: "encryptionkey@0"
I20250903 14:26:33.851598 20822 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 14:26:33.853683 20837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:33.854586 20822 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:33.854692 20822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "782c68d2503343918810c2bcbb6885b6"
format_stamp: "Formatted at 2025-09-03 14:26:33 on dist-test-slave-4wpb"
server_key: "ef60af3a724ae37ef1628051c602844b"
server_key_iv: "74fb7244bd82de0eca016b4df8444099"
server_key_version: "encryptionkey@0"
I20250903 14:26:33.854768 20822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:33.865450 20822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:33.867508 20822 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:33.867642 20822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:33.874416 20899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:42439 every 8 connection(s)
I20250903 14:26:33.874423 20822 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42439
I20250903 14:26:33.875245 20822 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:33.877404 20900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:33.881785 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Bootstrap starting.
I20250903 14:26:33.883001 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20822
I20250903 14:26:33.883085 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:33.883260 19718 external_mini_cluster.cc:1468] Setting key c54a85105860c954db48aa7bec28ae61
I20250903 14:26:33.883701 20900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:33.884472 20900 log.cc:826] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:33.886180 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: No bootstrap required, opened a new log
I20250903 14:26:33.888585 20900 raft_consensus.cc:357] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } }
I20250903 14:26:33.888777 20900 raft_consensus.cc:383] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:33.888842 20900 raft_consensus.cc:738] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782c68d2503343918810c2bcbb6885b6, State: Initialized, Role: FOLLOWER
I20250903 14:26:33.889292 20900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } }
I20250903 14:26:33.889439 20900 raft_consensus.cc:397] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:33.889511 20900 raft_consensus.cc:491] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:33.889604 20900 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 0 FOLLOWER]: Advancing to term 1
Sep 03 14:26:33 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:33.890326 20900 raft_consensus.cc:513] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } }
I20250903 14:26:33.890609 20900 leader_election.cc:304] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782c68d2503343918810c2bcbb6885b6; no voters:
I20250903 14:26:33.890846 20900 leader_election.cc:290] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:33.890937 20905 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:33.891155 20905 raft_consensus.cc:695] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [term 1 LEADER]: Becoming Leader. State: Replica: 782c68d2503343918810c2bcbb6885b6, State: Running, Role: LEADER
I20250903 14:26:33.891444 20905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } }
I20250903 14:26:33.891964 20900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:33.892428 20905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782c68d2503343918810c2bcbb6885b6. Latest consensus state: current_term: 1 leader_uuid: "782c68d2503343918810c2bcbb6885b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } } }
I20250903 14:26:33.892567 20905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:33.892860 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782c68d2503343918810c2bcbb6885b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782c68d2503343918810c2bcbb6885b6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42439 } } }
I20250903 14:26:33.893024 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:33.893316 20911 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:33.896037 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:33.884107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.19.65.190:42439)
0903 14:26:33.884453 (+ 346us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:33.884460 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:33.884490 (+ 30us) server_negotiation.cc:408] Connection header received
0903 14:26:33.885096 (+ 606us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:33.885121 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:33.885367 (+ 246us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:33.885632 (+ 265us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:33.886528 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:33.887236 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:33.887995 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:33.888157 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:33.890367 (+ 2210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:33.890382 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:33.890390 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:33.890416 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:33.893080 (+ 2664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:33.893589 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:33.893595 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:33.893599 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:33.893679 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:33.893945 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:33.893948 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:33.893950 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:33.894317 (+ 367us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:33.895072 (+ 755us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:33.895209 (+ 137us) server_negotiation.cc:300] Negotiation successful
0903 14:26:33.895414 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20250903 14:26:33.896060 20911 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:33.901270 20911 catalog_manager.cc:1349] Generated new cluster ID: 6413bf34add34c50ba6cba0f928889eb
I20250903 14:26:33.901330 20911 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:33.914315 20911 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:33.915239 20911 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:33.925700 20911 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Generated new TSK 0
I20250903 14:26:33.926182 20911 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:33.972481 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42439
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:34.061362 20928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:34.061549 20928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:34.061625 20928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:34.064399 20928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:34.064450 20928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:34.064522 20928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:34.064600 20928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:34.068060 20928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:34.068833 20928 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:34.069586 20928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:34.074745 20936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:34.074783 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:34.074745 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:34.074949 20928 server_base.cc:1047] running on GCE node
I20250903 14:26:34.075230 20928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:34.075623 20928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:34.076767 20928 hybrid_clock.cc:648] HybridClock initialized: now 1756909594076756 us; error 24 us; skew 500 ppm
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:34.079057 20928 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:34.079874 20928 webserver.cc:480] Webserver started at http://127.19.65.129:42485/ using document root <none> and password file <none>
I20250903 14:26:34.080368 20928 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:34.080415 20928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:34.080564 20928 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:34.082028 20928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "421965c563f90902081f56cc9e4ed51c"
server_key_iv: "3df316635f282f1540004f0af11426a9"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.082424 20928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "421965c563f90902081f56cc9e4ed51c"
server_key_iv: "3df316635f282f1540004f0af11426a9"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.085230 20928 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:26:34.087054 20943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.087844 20928 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:34.087941 20928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "421965c563f90902081f56cc9e4ed51c"
server_key_iv: "3df316635f282f1540004f0af11426a9"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.088011 20928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:34.098294 20928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:34.100214 20928 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:34.100350 20928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:34.101424 20928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:34.101490 20928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.101545 20928 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:34.101572 20928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:34.111176 20928 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40471
I20250903 14:26:34.111232 21058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:40471 every 8 connection(s)
I20250903 14:26:34.111804 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.102032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:41433 (local address 127.19.65.190:42439)
0903 14:26:34.102160 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.102164 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.102786 (+ 622us) server_negotiation.cc:408] Connection header received
0903 14:26:34.103689 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.103691 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.103737 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.103804 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.105252 (+ 1448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.105844 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.106629 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.106849 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.108607 (+ 1758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.108623 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.108625 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.108646 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.110095 (+ 1449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.110552 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.110555 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.110557 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.110599 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.110835 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.110848 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.110850 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.111001 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.111086 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.111562 (+ 476us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.111670 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 14:26:34.111868 20928 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:26:34.112313 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.102218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.129:41433)
0903 14:26:34.102653 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.102689 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.103495 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.103950 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.103962 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.104476 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.105113 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.105126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.105939 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.105944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.106499 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.106508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.106702 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.107222 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.107247 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.108475 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.110204 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.110209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.110217 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.110466 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.110675 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.110677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.110679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.110757 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.111129 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.111134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.111362 (+ 228us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.111516 (+ 154us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.111716 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
I20250903 14:26:34.113566 21059 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42439
I20250903 14:26:34.113705 21059 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:34.114147 21059 heartbeater.cc:507] Master 127.19.65.190:42439 requested a full tablet report, sending...
I20250903 14:26:34.115334 20853 ts_manager.cc:194] Registered new tserver with Master: 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129:40471)
I20250903 14:26:34.116357 20853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:41433
I20250903 14:26:34.116494 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 20928
I20250903 14:26:34.116576 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:34.116793 19718 external_mini_cluster.cc:1468] Setting key 68334fef49d3232822357ce6b464ff36
I20250903 14:26:34.121320 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.115147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:47131 (local address 127.19.65.190:42439)
0903 14:26:34.115254 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.115257 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.115446 (+ 189us) server_negotiation.cc:408] Connection header received
0903 14:26:34.115515 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.115518 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.115552 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.115608 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.116048 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.116473 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.117008 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.117425 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.118787 (+ 1362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.118803 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.118805 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.118823 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.120157 (+ 1334us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.120551 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.120553 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.120554 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.120590 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.120794 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.120796 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.120797 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.120888 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.120955 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.121165 (+ 210us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.121239 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:26:34.121447 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.115088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.129:47131)
0903 14:26:34.115349 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.115358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.115440 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.115628 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.115630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.115760 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.115931 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.115936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.116563 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.116567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.116916 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.116923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.117019 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.118423 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.118434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.118680 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.120254 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.120260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.120264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.120466 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.120665 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.120671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.120675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.120728 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.120954 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.120959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.121037 (+ 78us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.121128 (+ 91us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.121248 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:34.155575 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42439
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:34.243919 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:34.244079 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:34.244135 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:34.246860 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:34.246908 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:34.246951 21070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:34.246989 21070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:34.250286 21070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:34.250980 21070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:34.251590 21070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:34.256870 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:34.256867 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:34.256863 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:34.257086 21070 server_base.cc:1047] running on GCE node
I20250903 14:26:34.257422 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:34.257886 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:34.259019 21070 hybrid_clock.cc:648] HybridClock initialized: now 1756909594259010 us; error 27 us; skew 500 ppm
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:34.261132 21070 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:34.261911 21070 webserver.cc:480] Webserver started at http://127.19.65.130:39905/ using document root <none> and password file <none>
I20250903 14:26:34.262295 21070 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:34.262333 21070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:34.262449 21070 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:34.263761 21070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "89e9ee9e28f69f48f99391f5cce46e4f"
server_key_iv: "e176abf7ab4bb47297e64c3d023b1e66"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.264159 21070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "89e9ee9e28f69f48f99391f5cce46e4f"
server_key_iv: "e176abf7ab4bb47297e64c3d023b1e66"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.266924 21070 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250903 14:26:34.268620 21085 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.269376 21070 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:34.269457 21070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "89e9ee9e28f69f48f99391f5cce46e4f"
server_key_iv: "e176abf7ab4bb47297e64c3d023b1e66"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.269557 21070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:34.281340 21070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:34.283303 21070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:34.283443 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:34.284569 21070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:34.284610 21070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.284641 21070 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:34.284668 21070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:34.293982 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.285086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:55981 (local address 127.19.65.190:42439)
0903 14:26:34.285228 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.285232 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.285737 (+ 505us) server_negotiation.cc:408] Connection header received
0903 14:26:34.286413 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.286416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.286473 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.286538 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.287581 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.288068 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.288628 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.288802 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.290732 (+ 1930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.290747 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.290748 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.290770 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.292159 (+ 1389us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.292653 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.292657 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.292659 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.292741 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.293010 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.293014 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.293015 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.293140 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.293226 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.293712 (+ 486us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.293823 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250903 14:26:34.293995 21070 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:36509
I20250903 14:26:34.294036 21200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:36509 every 8 connection(s)
I20250903 14:26:34.294539 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.285269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.130:55981)
0903 14:26:34.285618 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.285651 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.286246 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.286673 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.286679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.287032 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.287452 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.287463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.288165 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.288168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.288528 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.288536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.288688 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.289278 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.289307 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.290615 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.292261 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.292266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.292275 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.292534 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.292816 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.292819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.292821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.292924 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.293235 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.293242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.293443 (+ 201us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.293659 (+ 216us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.293883 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20250903 14:26:34.294832 21070 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:26:34.296077 21201 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42439
I20250903 14:26:34.296211 21201 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:34.296617 21201 heartbeater.cc:507] Master 127.19.65.190:42439 requested a full tablet report, sending...
I20250903 14:26:34.297456 20853 ts_manager.cc:194] Registered new tserver with Master: 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509)
I20250903 14:26:34.298058 20853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:55981
I20250903 14:26:34.299497 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21070
I20250903 14:26:34.299854 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:34.300128 19718 external_mini_cluster.cc:1468] Setting key a3c3c4b402dcb562d3b9bbdfe6ce4465
I20250903 14:26:34.302701 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.297618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:32931 (local address 127.19.65.190:42439)
0903 14:26:34.297704 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.297706 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.297715 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:26:34.297739 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.297741 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.297770 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.297828 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.298331 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.298868 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.299424 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.299567 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.300260 (+ 693us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.300272 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.300274 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.300291 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.301431 (+ 1140us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.301849 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.301852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.301853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.301885 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.302097 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.302099 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.302100 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.302220 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.302285 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.302517 (+ 232us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.302603 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250903 14:26:34.302848 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.297332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.130:32931)
0903 14:26:34.297565 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.297577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.297646 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.297856 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.297858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.298005 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.298199 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.298206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.298994 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.298996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.299312 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.299321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.299421 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.299822 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.299837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.300167 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.301539 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.301544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.301548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.301769 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.301966 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.301971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.301976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.302031 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.302294 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.302299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.302380 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.302481 (+ 101us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.302619 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:34.339772 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42439
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:34.429337 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:34.429523 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:34.429582 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:34.432163 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:34.432204 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:34.432245 21212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:34.432278 21212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:34.435967 21212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:34.436676 21212 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:34.437304 21212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:34.442471 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:34.442541 21212 server_base.cc:1047] running on GCE node
W20250903 14:26:34.442451 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:34.442449 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:34.442972 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:34.443430 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:34.444582 21212 hybrid_clock.cc:648] HybridClock initialized: now 1756909594444572 us; error 24 us; skew 500 ppm
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:34.446745 21212 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:34.447530 21212 webserver.cc:480] Webserver started at http://127.19.65.131:45329/ using document root <none> and password file <none>
I20250903 14:26:34.447964 21212 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:34.448009 21212 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:34.448149 21212 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:34.449496 21212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "28989440cc9247c897100dafae777f4d"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "7e6923caca1e73bdd30c2d37fc89a837"
server_key_iv: "ee2b5dfc95409c6860e5e4030c012f40"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.449909 21212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "28989440cc9247c897100dafae777f4d"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "7e6923caca1e73bdd30c2d37fc89a837"
server_key_iv: "ee2b5dfc95409c6860e5e4030c012f40"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.452591 21212 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:34.454391 21227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.455190 21212 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:34.455279 21212 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "28989440cc9247c897100dafae777f4d"
format_stamp: "Formatted at 2025-09-03 14:26:34 on dist-test-slave-4wpb"
server_key: "7e6923caca1e73bdd30c2d37fc89a837"
server_key_iv: "ee2b5dfc95409c6860e5e4030c012f40"
server_key_version: "encryptionkey@0"
I20250903 14:26:34.455354 21212 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:34.468567 21212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:34.470662 21212 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:34.470774 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:34.471746 21212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:34.471788 21212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:34.471819 21212 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:34.471848 21212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:34.480160 21212 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:33361
I20250903 14:26:34.480182 21342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:33361 every 8 connection(s)
I20250903 14:26:34.480978 21212 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:34.482425 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.472437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.131:59651)
0903 14:26:34.472774 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.472807 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.473523 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.474020 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.474031 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.474471 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.475063 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.475075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.476093 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.476096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.476660 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.476672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.476843 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.477365 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.477403 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.478838 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.480624 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.480627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.480634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.480811 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.481077 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.481080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.481082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.481156 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.481547 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.481554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.481720 (+ 166us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.481837 (+ 117us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.481999 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20250903 14:26:34.482921 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.472248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59651 (local address 127.19.65.190:42439)
0903 14:26:34.472369 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.472373 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.472898 (+ 525us) server_negotiation.cc:408] Connection header received
0903 14:26:34.473709 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.473713 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.473779 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.473880 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.475215 (+ 1335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.475975 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.476790 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.477001 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.478977 (+ 1976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.478997 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.479001 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.479030 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.480537 (+ 1507us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.480905 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.480910 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.480914 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.480965 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.481239 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.481242 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.481244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.481401 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.482663 (+ 1262us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.482735 (+ 72us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.482820 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:26:34.483508 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21212
I20250903 14:26:34.483601 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:34.483675 21343 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42439
I20250903 14:26:34.483814 21343 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:34.483827 19718 external_mini_cluster.cc:1468] Setting key 544309e0e0345997f926071dd6a3821d
I20250903 14:26:34.484288 21343 heartbeater.cc:507] Master 127.19.65.190:42439 requested a full tablet report, sending...
I20250903 14:26:34.485306 20854 ts_manager.cc:194] Registered new tserver with Master: 28989440cc9247c897100dafae777f4d (127.19.65.131:33361)
I20250903 14:26:34.486035 20854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:59651
I20250903 14:26:34.490314 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.485578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:51931 (local address 127.19.65.190:42439)
0903 14:26:34.485674 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.485677 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.485957 (+ 280us) server_negotiation.cc:408] Connection header received
0903 14:26:34.486030 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.486032 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.486072 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.486148 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.486614 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.486987 (+ 373us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.487480 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.487648 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.488175 (+ 527us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.488188 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.488192 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.488210 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.489188 (+ 978us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.489520 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.489523 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.489525 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.489560 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.489753 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.489755 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.489757 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.489856 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.489937 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.490128 (+ 191us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.490206 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 14:26:34.490330 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.485688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.19.65.131:51931)
0903 14:26:34.485881 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.485891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.485958 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.486233 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.486236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.486361 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.486512 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.486516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.487073 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.487076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.487386 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.487392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.487485 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.487860 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.487870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.488073 (+ 203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.489279 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.489282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.489284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.489422 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.489645 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.489648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.489649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.489686 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.489939 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.489942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.490004 (+ 62us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.490081 (+ 77us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.490194 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20250903 14:26:34.496903 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:34.531900 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.525405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.19.65.190:42439)
0903 14:26:34.525561 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.525564 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.525574 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:26:34.525603 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.525605 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.525640 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.525705 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.526494 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.526947 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.527508 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.527698 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.529635 (+ 1937us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.529647 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.529649 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.529670 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.530690 (+ 1020us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.531010 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.531013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.531014 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.531049 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.531227 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.531230 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.531231 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.531344 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.531451 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.531649 (+ 198us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.531739 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 14:26:34.534312 20854 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:34.536592 20854 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:34.546540 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.543232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36509 (local address 127.0.0.1:38176)
0903 14:26:34.543516 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.543551 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.543666 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.544303 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.544308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.544333 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:34.544609 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.544616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.545611 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.545614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.546150 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.546157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546249 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.546291 (+ 42us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.546366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.546434 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20250903 14:26:34.547286 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.543422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.19.65.130:36509)
0903 14:26:34.543703 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.543707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.543728 (+ 21us) server_negotiation.cc:408] Connection header received
0903 14:26:34.543795 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.543799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.543940 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.544085 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:34.544736 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.545521 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.546270 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546742 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.547025 (+ 283us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.547076 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.547159 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20250903 14:26:34.547910 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.544202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33361 (local address 127.0.0.1:49760)
0903 14:26:34.544599 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.544612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.544683 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.545404 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.545406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.545423 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:34.545648 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.545655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546812 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.546815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.547575 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.547584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.547662 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.547674 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.547711 (+ 37us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.547759 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":79,"threads_started":1}
I20250903 14:26:34.547943 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.543883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40471 (local address 127.0.0.1:46660)
0903 14:26:34.544253 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.544269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.544351 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.545247 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.545250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.545268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:34.545465 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.545498 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546734 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.546738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.547575 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.547584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.547661 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.547674 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.547719 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.547759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20250903 14:26:34.548669 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.544036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.19.65.129:40471)
0903 14:26:34.544894 (+ 858us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.544899 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.544916 (+ 17us) server_negotiation.cc:408] Connection header received
0903 14:26:34.544974 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.544977 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.545116 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.545244 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:34.545628 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546610 (+ 982us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.547858 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.548341 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.548430 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.548483 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.548553 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":122,"threads_started":1}
I20250903 14:26:34.548971 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.544549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.19.65.131:33361)
0903 14:26:34.544875 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.544880 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.544901 (+ 21us) server_negotiation.cc:408] Connection header received
0903 14:26:34.544970 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.544974 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.545126 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.545244 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:34.545766 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.546717 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.548229 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.548643 (+ 414us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.548724 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.548783 (+ 59us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.548857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20250903 14:26:34.549533 21135 tablet_service.cc:1468] Processing CreateTablet for tablet f4e80bf17254404e8bb9d62299d49589 (DEFAULT_TABLE table=test-table [id=e782c17ab273464dab4eb4204332a66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.550390 21135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e80bf17254404e8bb9d62299d49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.550695 20993 tablet_service.cc:1468] Processing CreateTablet for tablet f4e80bf17254404e8bb9d62299d49589 (DEFAULT_TABLE table=test-table [id=e782c17ab273464dab4eb4204332a66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.550810 21277 tablet_service.cc:1468] Processing CreateTablet for tablet f4e80bf17254404e8bb9d62299d49589 (DEFAULT_TABLE table=test-table [id=e782c17ab273464dab4eb4204332a66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.551401 20993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e80bf17254404e8bb9d62299d49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.551517 21277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e80bf17254404e8bb9d62299d49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.555794 21366 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Bootstrap starting.
I20250903 14:26:34.556429 21367 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Bootstrap starting.
I20250903 14:26:34.556730 21368 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Bootstrap starting.
I20250903 14:26:34.557613 21366 tablet_bootstrap.cc:654] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.558306 21368 tablet_bootstrap.cc:654] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.559217 21366 log.cc:826] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:34.559374 21368 log.cc:826] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:34.559577 21367 tablet_bootstrap.cc:654] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.560591 21367 log.cc:826] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:34.560874 21366 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: No bootstrap required, opened a new log
I20250903 14:26:34.561038 21366 ts_tablet_manager.cc:1397] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250903 14:26:34.561738 21368 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: No bootstrap required, opened a new log
I20250903 14:26:34.561887 21368 ts_tablet_manager.cc:1397] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250903 14:26:34.562181 21367 tablet_bootstrap.cc:492] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: No bootstrap required, opened a new log
I20250903 14:26:34.562332 21367 ts_tablet_manager.cc:1397] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250903 14:26:34.563752 21366 raft_consensus.cc:357] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.563987 21366 raft_consensus.cc:383] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.564059 21366 raft_consensus.cc:738] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28989440cc9247c897100dafae777f4d, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.563948 21368 raft_consensus.cc:357] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.564194 21368 raft_consensus.cc:383] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.564275 21368 raft_consensus.cc:738] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725ca495fa3f4f8d9f9dff22a54bb5a1, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.564522 21366 consensus_queue.cc:260] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.564694 21368 consensus_queue.cc:260] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.565086 21367 raft_consensus.cc:357] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.565201 21343 heartbeater.cc:499] Master 127.19.65.190:42439 was elected leader, sending a full tablet report...
I20250903 14:26:34.565304 21367 raft_consensus.cc:383] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.565382 21368 ts_tablet_manager.cc:1428] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 14:26:34.565389 21367 raft_consensus.cc:738] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc6db8b64ac4c51a4fb6e7b0ab8de3f, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.565400 21366 ts_tablet_manager.cc:1428] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:26:34.565768 21367 consensus_queue.cc:260] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.565855 21059 heartbeater.cc:499] Master 127.19.65.190:42439 was elected leader, sending a full tablet report...
I20250903 14:26:34.566298 21201 heartbeater.cc:499] Master 127.19.65.190:42439 was elected leader, sending a full tablet report...
I20250903 14:26:34.566422 21367 ts_tablet_manager.cc:1428] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250903 14:26:34.612854 21060 tablet.cc:2378] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:26:34.672600 21373 raft_consensus.cc:491] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:34.672791 21373 raft_consensus.cc:513] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.673925 21373 leader_election.cc:290] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509), 28989440cc9247c897100dafae777f4d (127.19.65.131:33361)
I20250903 14:26:34.676946 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.674501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33361 (local address 127.19.65.129:41811)
0903 14:26:34.674719 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.674731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.674829 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.675039 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.675042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.675056 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:34.675262 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.675268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.675922 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.675926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.676514 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.676522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.676628 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.676650 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.676702 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.676759 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20250903 14:26:34.676954 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.674115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36509 (local address 127.19.65.129:33159)
0903 14:26:34.674429 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.674445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.674568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.675026 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.675029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.675047 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:34.675229 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.675235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.675897 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.675900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.676524 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.676532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.676663 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.676683 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.676737 (+ 54us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.676794 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20250903 14:26:34.677397 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.674629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:41811 (local address 127.19.65.131:33361)
0903 14:26:34.674739 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.674745 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.674783 (+ 38us) server_negotiation.cc:408] Connection header received
0903 14:26:34.674917 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.674920 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.674964 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.675023 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:34.675361 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.675783 (+ 422us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.676628 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.677125 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.677165 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.677221 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.677290 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:26:34.677455 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.674222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33159 (local address 127.19.65.130:36509)
0903 14:26:34.674378 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.674383 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.674500 (+ 117us) server_negotiation.cc:408] Connection header received
0903 14:26:34.674902 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.674905 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.674947 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.675007 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:34.675329 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.675764 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.676658 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.677159 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.677204 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.677254 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.677319 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 14:26:34.677989 21279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4e80bf17254404e8bb9d62299d49589" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28989440cc9247c897100dafae777f4d" is_pre_election: true
I20250903 14:26:34.678004 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4e80bf17254404e8bb9d62299d49589" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" is_pre_election: true
I20250903 14:26:34.678198 21279 raft_consensus.cc:2466] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 0.
I20250903 14:26:34.678221 21155 raft_consensus.cc:2466] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 0.
I20250903 14:26:34.678583 20945 leader_election.cc:304] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28989440cc9247c897100dafae777f4d, 725ca495fa3f4f8d9f9dff22a54bb5a1; no voters:
I20250903 14:26:34.678766 21373 raft_consensus.cc:2802] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:34.678823 21373 raft_consensus.cc:491] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:34.678849 21373 raft_consensus.cc:3058] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.679636 21373 raft_consensus.cc:513] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.679922 21373 leader_election.cc:290] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 election: Requested vote from peers 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509), 28989440cc9247c897100dafae777f4d (127.19.65.131:33361)
I20250903 14:26:34.680254 21279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4e80bf17254404e8bb9d62299d49589" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28989440cc9247c897100dafae777f4d"
I20250903 14:26:34.680260 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4e80bf17254404e8bb9d62299d49589" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
I20250903 14:26:34.680343 21279 raft_consensus.cc:3058] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.680352 21155 raft_consensus.cc:3058] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.681237 21279 raft_consensus.cc:2466] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 1.
I20250903 14:26:34.681265 21155 raft_consensus.cc:2466] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 1.
I20250903 14:26:34.681516 20945 leader_election.cc:304] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28989440cc9247c897100dafae777f4d, 725ca495fa3f4f8d9f9dff22a54bb5a1; no voters:
I20250903 14:26:34.681654 21373 raft_consensus.cc:2802] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:34.681810 21373 raft_consensus.cc:695] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 725ca495fa3f4f8d9f9dff22a54bb5a1, State: Running, Role: LEADER
I20250903 14:26:34.682085 21373 consensus_queue.cc:237] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.684728 20852 catalog_manager.cc:5582] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } health_report { overall_health: UNKNOWN } } }
I20250903 14:26:34.696980 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.694666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.19.65.129:40471)
0903 14:26:34.694761 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.694764 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.694774 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:26:34.694908 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.694911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.694950 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.695014 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:34.695246 (+ 232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.695621 (+ 375us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.696177 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.696286 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.696323 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:34.696683 (+ 360us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:34.696762 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.696866 (+ 104us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.696904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:26:34.700979 21279 raft_consensus.cc:1273] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Refusing update from remote peer 725ca495fa3f4f8d9f9dff22a54bb5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:34.701014 21155 raft_consensus.cc:1273] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Refusing update from remote peer 725ca495fa3f4f8d9f9dff22a54bb5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:34.701429 21373 consensus_queue.cc:1035] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:34.701578 21377 consensus_queue.cc:1035] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:34.706768 21380 mvcc.cc:204] Tried to move back new op lower bound from 7196301699890753536 to 7196301699820109824. Current Snapshot: MvccSnapshot[applied={T|T < 7196301699890753536}]
I20250903 14:26:34.708187 21382 mvcc.cc:204] Tried to move back new op lower bound from 7196301699890753536 to 7196301699820109824. Current Snapshot: MvccSnapshot[applied={T|T < 7196301699890753536}]
I20250903 14:26:34.715667 20852 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:34.715794 20852 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:34.717782 20852 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Sending DeleteTablet for 3 replicas of tablet f4e80bf17254404e8bb9d62299d49589
I20250903 14:26:34.718308 21135 tablet_service.cc:1515] Processing DeleteTablet for tablet f4e80bf17254404e8bb9d62299d49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:34 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:38176
I20250903 14:26:34.718385 21277 tablet_service.cc:1515] Processing DeleteTablet for tablet f4e80bf17254404e8bb9d62299d49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:34 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:49760
I20250903 14:26:34.718380 20993 tablet_service.cc:1515] Processing DeleteTablet for tablet f4e80bf17254404e8bb9d62299d49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:34 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46660
I20250903 14:26:34.718741 21391 tablet_replica.cc:331] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: stopping tablet replica
I20250903 14:26:34.718748 21389 tablet_replica.cc:331] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: stopping tablet replica
I20250903 14:26:34.718948 21390 tablet_replica.cc:331] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: stopping tablet replica
I20250903 14:26:34.719079 21391 raft_consensus.cc:2241] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:34.719162 21390 raft_consensus.cc:2241] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:34.719415 21390 raft_consensus.cc:2270] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:34.719415 21391 raft_consensus.cc:2270] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:34.719383 20852 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:34.719717 20852 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:34.720584 21390 ts_tablet_manager.cc:1905] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:34.720583 21391 ts_tablet_manager.cc:1905] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:34.722664 21390 ts_tablet_manager.cc:1918] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:26:34.722672 21391 ts_tablet_manager.cc:1918] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:26:34.722740 21391 log.cc:1199] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/f4e80bf17254404e8bb9d62299d49589
I20250903 14:26:34.722766 21390 log.cc:1199] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/f4e80bf17254404e8bb9d62299d49589
I20250903 14:26:34.723031 21391 ts_tablet_manager.cc:1939] T f4e80bf17254404e8bb9d62299d49589 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting consensus metadata
I20250903 14:26:34.723042 21390 ts_tablet_manager.cc:1939] T f4e80bf17254404e8bb9d62299d49589 P 28989440cc9247c897100dafae777f4d: Deleting consensus metadata
I20250903 14:26:34.724020 20839 catalog_manager.cc:4928] TS 28989440cc9247c897100dafae777f4d (127.19.65.131:33361): tablet f4e80bf17254404e8bb9d62299d49589 (table test-table [id=e782c17ab273464dab4eb4204332a66b]) successfully deleted
I20250903 14:26:34.724664 21135 tablet_service.cc:1468] Processing CreateTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 (DEFAULT_TABLE table=test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.724712 20993 tablet_service.cc:1468] Processing CreateTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 (DEFAULT_TABLE table=test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.724875 21389 raft_consensus.cc:2241] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:34.724910 20993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c1b28d8f7445d9c2d2c3e00dff070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.724912 21135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c1b28d8f7445d9c2d2c3e00dff070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.725126 21389 raft_consensus.cc:2270] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:34.726824 21368 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Bootstrap starting.
I20250903 14:26:34.726897 21367 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Bootstrap starting.
I20250903 14:26:34.727083 21277 tablet_service.cc:1468] Processing CreateTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 (DEFAULT_TABLE table=test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:34.727293 21277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c1b28d8f7445d9c2d2c3e00dff070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.727711 21367 tablet_bootstrap.cc:654] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.727711 21368 tablet_bootstrap.cc:654] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.728775 21366 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Bootstrap starting.
I20250903 14:26:34.728895 21368 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: No bootstrap required, opened a new log
I20250903 14:26:34.728965 21368 ts_tablet_manager.cc:1397] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:34.729084 20839 catalog_manager.cc:4928] TS 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129:40471): tablet f4e80bf17254404e8bb9d62299d49589 (table test-table [id=e782c17ab273464dab4eb4204332a66b]) successfully deleted
I20250903 14:26:34.729394 21368 raft_consensus.cc:357] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.729508 21368 raft_consensus.cc:383] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.729542 21368 raft_consensus.cc:738] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725ca495fa3f4f8d9f9dff22a54bb5a1, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.729664 21366 tablet_bootstrap.cc:654] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.729640 21368 consensus_queue.cc:260] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.729892 21368 ts_tablet_manager.cc:1428] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:26:34.730482 21389 ts_tablet_manager.cc:1905] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:34.731559 21366 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: No bootstrap required, opened a new log
I20250903 14:26:34.731638 21366 ts_tablet_manager.cc:1397] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:26:34.732070 21366 raft_consensus.cc:357] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.732172 21366 raft_consensus.cc:383] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.732199 21366 raft_consensus.cc:738] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28989440cc9247c897100dafae777f4d, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.732335 21389 ts_tablet_manager.cc:1918] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:26:34.732303 21366 consensus_queue.cc:260] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.732398 21389 log.cc:1199] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/f4e80bf17254404e8bb9d62299d49589
I20250903 14:26:34.732527 21366 ts_tablet_manager.cc:1428] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:34.732652 21389 ts_tablet_manager.cc:1939] T f4e80bf17254404e8bb9d62299d49589 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting consensus metadata
I20250903 14:26:34.733680 20839 catalog_manager.cc:4928] TS 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509): tablet f4e80bf17254404e8bb9d62299d49589 (table test-table [id=e782c17ab273464dab4eb4204332a66b]) successfully deleted
I20250903 14:26:34.734217 21367 tablet_bootstrap.cc:492] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: No bootstrap required, opened a new log
I20250903 14:26:34.734299 21367 ts_tablet_manager.cc:1397] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250903 14:26:34.734719 21367 raft_consensus.cc:357] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.734807 21367 raft_consensus.cc:383] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.734841 21367 raft_consensus.cc:738] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc6db8b64ac4c51a4fb6e7b0ab8de3f, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.734944 21367 consensus_queue.cc:260] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.735157 21367 ts_tablet_manager.cc:1428] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250903 14:26:34.795699 21202 tablet.cc:2378] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:26:34.806636 21377 raft_consensus.cc:491] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:34.806730 21377 raft_consensus.cc:513] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.807085 21377 leader_election.cc:290] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509), 28989440cc9247c897100dafae777f4d (127.19.65.131:33361)
I20250903 14:26:34.807497 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51c1b28d8f7445d9c2d2c3e00dff070" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" is_pre_election: true
I20250903 14:26:34.807566 21279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51c1b28d8f7445d9c2d2c3e00dff070" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28989440cc9247c897100dafae777f4d" is_pre_election: true
I20250903 14:26:34.807619 21155 raft_consensus.cc:2466] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 0.
I20250903 14:26:34.807710 21279 raft_consensus.cc:2466] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 0.
I20250903 14:26:34.807920 20945 leader_election.cc:304] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc6db8b64ac4c51a4fb6e7b0ab8de3f, 725ca495fa3f4f8d9f9dff22a54bb5a1; no voters:
I20250903 14:26:34.808089 21377 raft_consensus.cc:2802] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:34.808147 21377 raft_consensus.cc:491] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:34.808204 21377 raft_consensus.cc:3058] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.809021 21377 raft_consensus.cc:513] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.809320 21377 leader_election.cc:290] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 election: Requested vote from peers 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509), 28989440cc9247c897100dafae777f4d (127.19.65.131:33361)
I20250903 14:26:34.809635 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51c1b28d8f7445d9c2d2c3e00dff070" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
I20250903 14:26:34.809672 21279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51c1b28d8f7445d9c2d2c3e00dff070" candidate_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28989440cc9247c897100dafae777f4d"
I20250903 14:26:34.809733 21155 raft_consensus.cc:3058] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.809755 21279 raft_consensus.cc:3058] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:34.810595 21279 raft_consensus.cc:2466] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 1.
I20250903 14:26:34.810595 21155 raft_consensus.cc:2466] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725ca495fa3f4f8d9f9dff22a54bb5a1 in term 1.
I20250903 14:26:34.810839 20945 leader_election.cc:304] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28989440cc9247c897100dafae777f4d, 725ca495fa3f4f8d9f9dff22a54bb5a1; no voters:
I20250903 14:26:34.811041 21377 raft_consensus.cc:2802] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:34.811112 21377 raft_consensus.cc:695] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 725ca495fa3f4f8d9f9dff22a54bb5a1, State: Running, Role: LEADER
I20250903 14:26:34.811226 21377 consensus_queue.cc:237] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.812492 20853 catalog_manager.cc:5582] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } health_report { overall_health: UNKNOWN } } }
Sep 03 14:26:34 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909593, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:34.854158 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.846866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.19.65.190:42439)
0903 14:26:34.846975 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:34.846979 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:34.847099 (+ 120us) server_negotiation.cc:408] Connection header received
0903 14:26:34.847177 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:34.847180 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:34.847221 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:34.847290 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:34.847946 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.848652 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.849174 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.849356 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.851313 (+ 1957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:34.851327 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:34.851329 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:34.851350 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:34.852663 (+ 1313us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.853128 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.853130 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.853131 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.853161 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:34.853353 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:34.853355 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:34.853356 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:34.853505 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:34.853586 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:34.853857 (+ 271us) server_negotiation.cc:300] Negotiation successful
0903 14:26:34.854017 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":43}
I20250903 14:26:34.854171 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:34.846794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42439 (local address 127.0.0.1:58662)
0903 14:26:34.847014 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:34.847026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:34.847103 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:34.847311 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:34.847313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:34.847647 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:34.847821 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.847839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.848747 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:34.848750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:34.849078 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:34.849083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:34.849157 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:34.849723 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:34.849735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:34.851193 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:34.852762 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.852770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.852787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.853047 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.853223 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:34.853225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:34.853228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:34.853273 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:34.853597 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:34.853603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:34.853710 (+ 107us) client_negotiation.cc:769] Sending connection context
0903 14:26:34.853823 (+ 113us) client_negotiation.cc:240] Negotiation successful
0903 14:26:34.853958 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20250903 14:26:34.857620 20853 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:58662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 14:26:34.858242 20853 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:34.862486 20993 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78484c8fb146b4a5abde6abd714256 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d7ae3f46f54ce5b9722841d750c26a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:34.862484 21135 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78484c8fb146b4a5abde6abd714256 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d7ae3f46f54ce5b9722841d750c26a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:34.862794 20993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78484c8fb146b4a5abde6abd714256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.862794 21135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78484c8fb146b4a5abde6abd714256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.862716 21277 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78484c8fb146b4a5abde6abd714256 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d7ae3f46f54ce5b9722841d750c26a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:34.863044 21277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78484c8fb146b4a5abde6abd714256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:34.865253 21368 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Bootstrap starting.
I20250903 14:26:34.865756 21366 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: Bootstrap starting.
I20250903 14:26:34.866051 21368 tablet_bootstrap.cc:654] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.866672 21367 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Bootstrap starting.
I20250903 14:26:34.866688 21366 tablet_bootstrap.cc:654] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.867511 21367 tablet_bootstrap.cc:654] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:34.867822 21366 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: No bootstrap required, opened a new log
I20250903 14:26:34.867915 21366 ts_tablet_manager.cc:1397] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:34.868427 21368 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: No bootstrap required, opened a new log
I20250903 14:26:34.868429 21366 raft_consensus.cc:357] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.868520 21368 ts_tablet_manager.cc:1397] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250903 14:26:34.868535 21366 raft_consensus.cc:383] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.868568 21366 raft_consensus.cc:738] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28989440cc9247c897100dafae777f4d, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.868695 21366 consensus_queue.cc:260] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.868957 21366 ts_tablet_manager.cc:1428] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:34.868942 21368 raft_consensus.cc:357] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.868925 21372 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:34.869035 21368 raft_consensus.cc:383] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.869050 21372 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:34.869071 21368 raft_consensus.cc:738] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725ca495fa3f4f8d9f9dff22a54bb5a1, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.869186 21368 consensus_queue.cc:260] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.869488 21368 ts_tablet_manager.cc:1428] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:34.869685 21377 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:34.869794 21377 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:34.869804 21367 tablet_bootstrap.cc:492] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: No bootstrap required, opened a new log
I20250903 14:26:34.869872 21367 ts_tablet_manager.cc:1397] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:26:34.870322 21367 raft_consensus.cc:357] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.870417 21367 raft_consensus.cc:383] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:34.870445 21367 raft_consensus.cc:738] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc6db8b64ac4c51a4fb6e7b0ab8de3f, State: Initialized, Role: FOLLOWER
I20250903 14:26:34.870545 21367 consensus_queue.cc:260] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:34.870790 21367 ts_tablet_manager.cc:1428] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:34.871024 21374 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:34.871130 21374 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: This TxnStatusTablet replica's current role is: FOLLOWER
W20250903 14:26:34.981848 21344 tablet.cc:2378] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:26:35.052186 21372 raft_consensus.cc:491] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:35.052310 21372 raft_consensus.cc:513] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:35.053218 21372 leader_election.cc:290] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129:40471), 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509)
I20250903 14:26:35.056277 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.053409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40471 (local address 127.19.65.131:38725)
0903 14:26:35.053670 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:35.053687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:35.053793 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:35.054070 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:35.054074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:35.054094 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:35.054319 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.054326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.055161 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.055163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:35.055888 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.055897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.055998 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.056019 (+ 21us) client_negotiation.cc:769] Sending connection context
0903 14:26:35.056068 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 14:26:35.056131 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"spinlock_wait_cycles":3328,"thread_start_us":96,"threads_started":1}
I20250903 14:26:35.056298 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.053776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36509 (local address 127.19.65.131:50391)
0903 14:26:35.054017 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:35.054027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:35.054112 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:35.054353 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:35.054356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:35.054372 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:35.054609 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.054615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.055329 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.055332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:35.056004 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.056011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.056117 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.056127 (+ 10us) client_negotiation.cc:769] Sending connection context
0903 14:26:35.056169 (+ 42us) client_negotiation.cc:240] Negotiation successful
0903 14:26:35.056210 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":55,"threads_started":1}
I20250903 14:26:35.056746 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.053867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:50391 (local address 127.19.65.130:36509)
0903 14:26:35.053986 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.053992 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.054104 (+ 112us) server_negotiation.cc:408] Connection header received
0903 14:26:35.054225 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.054227 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.054270 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.054333 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:35.054699 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.055256 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.056122 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.056532 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.056564 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.056603 (+ 39us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.056652 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 14:26:35.056746 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.053523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38725 (local address 127.19.65.129:40471)
0903 14:26:35.053642 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.053646 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.053764 (+ 118us) server_negotiation.cc:408] Connection header received
0903 14:26:35.053910 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.053917 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.053969 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.054067 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:35.054444 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.055066 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.056020 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.056497 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.056533 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.056582 (+ 49us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.056642 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:26:35.057137 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78484c8fb146b4a5abde6abd714256" candidate_uuid: "28989440cc9247c897100dafae777f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" is_pre_election: true
I20250903 14:26:35.057207 21013 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78484c8fb146b4a5abde6abd714256" candidate_uuid: "28989440cc9247c897100dafae777f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" is_pre_election: true
I20250903 14:26:35.057263 21155 raft_consensus.cc:2466] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28989440cc9247c897100dafae777f4d in term 0.
I20250903 14:26:35.057390 21013 raft_consensus.cc:2466] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28989440cc9247c897100dafae777f4d in term 0.
I20250903 14:26:35.057595 21229 leader_election.cc:304] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28989440cc9247c897100dafae777f4d, 3cc6db8b64ac4c51a4fb6e7b0ab8de3f; no voters:
I20250903 14:26:35.057794 21372 raft_consensus.cc:2802] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:35.057845 21372 raft_consensus.cc:491] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:35.057894 21372 raft_consensus.cc:3058] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:35.058645 21372 raft_consensus.cc:513] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:35.058909 21372 leader_election.cc:290] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [CANDIDATE]: Term 1 election: Requested vote from peers 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129:40471), 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509)
I20250903 14:26:35.059244 21155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78484c8fb146b4a5abde6abd714256" candidate_uuid: "28989440cc9247c897100dafae777f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f"
I20250903 14:26:35.059260 21013 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78484c8fb146b4a5abde6abd714256" candidate_uuid: "28989440cc9247c897100dafae777f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1"
I20250903 14:26:35.059338 21013 raft_consensus.cc:3058] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:35.059360 21155 raft_consensus.cc:3058] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:35.060065 21013 raft_consensus.cc:2466] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28989440cc9247c897100dafae777f4d in term 1.
I20250903 14:26:35.060066 21155 raft_consensus.cc:2466] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28989440cc9247c897100dafae777f4d in term 1.
I20250903 14:26:35.060290 21229 leader_election.cc:304] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28989440cc9247c897100dafae777f4d, 725ca495fa3f4f8d9f9dff22a54bb5a1; no voters:
I20250903 14:26:35.060446 21372 raft_consensus.cc:2802] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:35.060573 21372 raft_consensus.cc:695] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [term 1 LEADER]: Becoming Leader. State: Replica: 28989440cc9247c897100dafae777f4d, State: Running, Role: LEADER
I20250903 14:26:35.060781 21372 consensus_queue.cc:237] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } }
I20250903 14:26:35.061556 21407 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: TxnStatusTablet state changed. Reason: New leader 28989440cc9247c897100dafae777f4d. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.061642 21407 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:35.061892 21409 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 14:26:35.062021 21409 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 14:26:35.062638 20852 catalog_manager.cc:5582] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 28989440cc9247c897100dafae777f4d (127.19.65.131). New cstate: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } health_report { overall_health: HEALTHY } } }
I20250903 14:26:35.107358 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.104857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.19.65.131:33361)
0903 14:26:35.104962 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.104967 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.104978 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:26:35.105121 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.105124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.105166 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.105244 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:35.105529 (+ 285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.105920 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.106473 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.106642 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.106684 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.107011 (+ 327us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.107110 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.107213 (+ 103us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.107255 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250903 14:26:35.107360 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.104761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33361 (local address 127.0.0.1:49776)
0903 14:26:35.104897 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:35.104910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:35.105010 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:35.105259 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:35.105262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:35.105269 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:35.105426 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.105431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.106017 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.106020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:35.106369 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.106375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.106471 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.106509 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.107111 (+ 602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.107126 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:26:35.107189 (+ 63us) client_negotiation.cc:240] Negotiation successful
0903 14:26:35.107258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20250903 14:26:35.109447 21155 raft_consensus.cc:1273] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Refusing update from remote peer 28989440cc9247c897100dafae777f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:35.109674 21013 raft_consensus.cc:1273] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Refusing update from remote peer 28989440cc9247c897100dafae777f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:35.109889 21407 consensus_queue.cc:1035] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:35.110065 21372 consensus_queue.cc:1035] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:35.111198 21374 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: TxnStatusTablet state changed. Reason: New leader 28989440cc9247c897100dafae777f4d. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.111299 21374 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:35.111479 21377 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: TxnStatusTablet state changed. Reason: New leader 28989440cc9247c897100dafae777f4d. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.111570 21377 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:35.113266 21372 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.113382 21372 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:35.114009 21381 mvcc.cc:204] Tried to move back new op lower bound from 7196301701564747776 to 7196301701370871808. Current Snapshot: MvccSnapshot[applied={T|T < 7196301701564747776 or (T in {7196301701564747776})}]
I20250903 14:26:35.114611 21374 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.114709 21374 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:35.116519 21372 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.116603 21372 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 28989440cc9247c897100dafae777f4d: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:35.118085 21377 tablet_replica.cc:440] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28989440cc9247c897100dafae777f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725ca495fa3f4f8d9f9dff22a54bb5a1" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40471 } } peers { permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 } } peers { permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 } } }
I20250903 14:26:35.118175 21377 tablet_replica.cc:443] T 3f78484c8fb146b4a5abde6abd714256 P 725ca495fa3f4f8d9f9dff22a54bb5a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:35.127657 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.125183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:51735 (local address 127.19.65.131:33361)
0903 14:26:35.125267 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.125271 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.125285 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:26:35.125319 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.125322 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.125357 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.125425 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:35.125803 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.126377 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.127145 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.127299 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.127332 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.127386 (+ 54us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.127450 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.127531 (+ 81us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.127565 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250903 14:26:35.127691 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.124883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33361 (local address 127.19.65.129:51735)
0903 14:26:35.125159 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:35.125173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:35.125253 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:35.125447 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:35.125450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:35.125457 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:35.125704 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.125710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.126476 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.126479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:35.127033 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.127040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.127137 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.127172 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.127460 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.127464 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 14:26:35.127519 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 14:26:35.127578 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20250903 14:26:35.133703 21279 raft_consensus.cc:1273] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Refusing update from remote peer 725ca495fa3f4f8d9f9dff22a54bb5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:35.133780 21155 raft_consensus.cc:1273] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Refusing update from remote peer 725ca495fa3f4f8d9f9dff22a54bb5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:35.134095 21373 consensus_queue.cc:1035] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28989440cc9247c897100dafae777f4d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:35.134199 21377 consensus_queue.cc:1035] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc6db8b64ac4c51a4fb6e7b0ab8de3f" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:35.153050 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.149418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40471 (local address 127.19.65.131:38659)
0903 14:26:35.150610 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:35.150621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:35.150707 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:35.150891 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:35.150893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:35.150900 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:35.151142 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.151149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.152031 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.152033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:35.152444 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:35.152450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.152527 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.152553 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.152870 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.152874 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 14:26:35.152915 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:26:35.152958 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":155,"threads_started":1}
I20250903 14:26:35.153071 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.149458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38659 (local address 127.19.65.129:40471)
0903 14:26:35.149589 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.149618 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.150683 (+ 1065us) server_negotiation.cc:408] Connection header received
0903 14:26:35.150774 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.150778 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.150816 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.150877 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:35.151281 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.151932 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.152533 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.152693 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.152729 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:35.152792 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:35.152866 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.152939 (+ 73us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.152978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:26:35.170527 20852 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:35.170650 20852 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:35.172410 20852 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 782c68d2503343918810c2bcbb6885b6: Sending DeleteTablet for 3 replicas of tablet f51c1b28d8f7445d9c2d2c3e00dff070
I20250903 14:26:35.173097 21135 tablet_service.cc:1515] Processing DeleteTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:38176
I20250903 14:26:35.173233 21389 tablet_replica.cc:331] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: stopping tablet replica
I20250903 14:26:35.173342 21389 raft_consensus.cc:2241] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:35.173457 21389 raft_consensus.cc:2270] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:35.173755 21277 tablet_service.cc:1515] Processing DeleteTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:49760
I20250903 14:26:35.173940 21390 tablet_replica.cc:331] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: stopping tablet replica
I20250903 14:26:35.174048 21390 raft_consensus.cc:2241] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:35.174144 21390 raft_consensus.cc:2270] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:35.174814 21390 ts_tablet_manager.cc:1905] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:35.175098 21389 ts_tablet_manager.cc:1905] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:35.177161 21390 ts_tablet_manager.cc:1918] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:35.177253 21390 log.cc:1199] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/f51c1b28d8f7445d9c2d2c3e00dff070
I20250903 14:26:35.177340 21389 ts_tablet_manager.cc:1918] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:35.177395 21389 log.cc:1199] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/f51c1b28d8f7445d9c2d2c3e00dff070
I20250903 14:26:35.177500 21390 ts_tablet_manager.cc:1939] T f51c1b28d8f7445d9c2d2c3e00dff070 P 28989440cc9247c897100dafae777f4d: Deleting consensus metadata
I20250903 14:26:35.177623 21389 ts_tablet_manager.cc:1939] T f51c1b28d8f7445d9c2d2c3e00dff070 P 3cc6db8b64ac4c51a4fb6e7b0ab8de3f: Deleting consensus metadata
I20250903 14:26:35.178058 20839 catalog_manager.cc:4928] TS 28989440cc9247c897100dafae777f4d (127.19.65.131:33361): tablet f51c1b28d8f7445d9c2d2c3e00dff070 (table test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]) successfully deleted
I20250903 14:26:35.178369 20993 tablet_service.cc:1515] Processing DeleteTablet for tablet f51c1b28d8f7445d9c2d2c3e00dff070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46660
I20250903 14:26:35.178535 21391 tablet_replica.cc:331] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: stopping tablet replica
I20250903 14:26:35.178627 21391 raft_consensus.cc:2241] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:35.178776 21391 raft_consensus.cc:2270] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:35.178936 20839 catalog_manager.cc:4928] TS 3cc6db8b64ac4c51a4fb6e7b0ab8de3f (127.19.65.130:36509): tablet f51c1b28d8f7445d9c2d2c3e00dff070 (table test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]) successfully deleted
I20250903 14:26:35.179373 21391 ts_tablet_manager.cc:1905] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:35.181311 21391 ts_tablet_manager.cc:1918] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:35.181371 21391 log.cc:1199] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/f51c1b28d8f7445d9c2d2c3e00dff070
I20250903 14:26:35.181615 21391 ts_tablet_manager.cc:1939] T f51c1b28d8f7445d9c2d2c3e00dff070 P 725ca495fa3f4f8d9f9dff22a54bb5a1: Deleting consensus metadata
I20250903 14:26:35.182097 20839 catalog_manager.cc:4928] TS 725ca495fa3f4f8d9f9dff22a54bb5a1 (127.19.65.129:40471): tablet f51c1b28d8f7445d9c2d2c3e00dff070 (table test-table [id=71002ef9ac184dd0937aaf9dd79c24f2]) successfully deleted
Sep 03 14:26:35 dist-test-slave-4wpb krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:35.186313 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.178217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.19.65.129:40471)
0903 14:26:35.178331 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.178335 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.178348 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:26:35.178377 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.178380 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.178416 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.178485 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:35.179522 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.180049 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.180825 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.180993 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.183362 (+ 2369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:35.183380 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:35.183388 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:35.183409 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:35.185042 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:35.185404 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:35.185407 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:35.185408 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:35.185447 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:35.185660 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:35.185662 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:35.185663 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:35.185819 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:35.185947 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.186129 (+ 182us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.186206 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20250903 14:26:35.186705 21053 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46678
I20250903 14:26:35.194186 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:35.188060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.19.65.190:42439)
0903 14:26:35.188168 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:35.188171 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:35.188268 (+ 97us) server_negotiation.cc:408] Connection header received
0903 14:26:35.188365 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:35.188367 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:35.188399 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:35.188454 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:35.189217 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.189645 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:35.190320 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:35.190436 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:35.191477 (+ 1041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:35.191495 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:35.191500 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:35.191529 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:35.192738 (+ 1209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:35.193158 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:35.193160 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:35.193161 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:35.193196 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:35.193432 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:35.193434 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:35.193435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:35.193594 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:35.193691 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:35.193980 (+ 289us) server_negotiation.cc:300] Negotiation successful
0903 14:26:35.194069 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20250903 14:26:35.194518 20852 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58676
I20250903 14:26:35.195515 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20928
I20250903 14:26:35.203403 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21070
I20250903 14:26:35.209061 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21212
I20250903 14:26:35.214994 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 20822
2025-09-03T14:26:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3385 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:35 dist-test-slave-4wpb krb5kdc[21430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:35 dist-test-slave-4wpb krb5kdc[21430](info): set up 2 sockets
Sep 03 14:26:35 dist-test-slave-4wpb krb5kdc[21430](info): commencing operation
krb5kdc: starting...
W20250903 14:26:36.965374 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.729s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:37.073803 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39553
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:37.160844 21446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:37.161016 21446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:37.161092 21446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:37.163745 21446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:37.163797 21446 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 14:26:37.163829 21446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:37.163866 21446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:37.163904 21446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:37.163944 21446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:37.167411 21446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39553
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:37.168262 21446 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:37.168948 21446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:37.173530 21454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.173539 21452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:37.173671 21446 server_base.cc:1047] running on GCE node
W20250903 14:26:37.173593 21451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:37.174212 21446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:37.174937 21446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:37.176079 21446 hybrid_clock.cc:648] HybridClock initialized: now 1756909597176067 us; error 24 us; skew 500 ppm
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:37.178231 21446 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:37.179083 21446 webserver.cc:480] Webserver started at http://127.19.65.190:38767/ using document root <none> and password file <none>
I20250903 14:26:37.179503 21446 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:37.179545 21446 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:37.179702 21446 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:37.181118 21446 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "9eae0607571245ceba488d7aeaa520c1"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "a961c826e776689b3f78b13e381366a1"
server_key_iv: "495ca9cd313f277834ca8c9242c5d694"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.181542 21446 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "9eae0607571245ceba488d7aeaa520c1"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "a961c826e776689b3f78b13e381366a1"
server_key_iv: "495ca9cd313f277834ca8c9242c5d694"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.184278 21446 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250903 14:26:37.186002 21461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.186774 21446 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:26:37.186856 21446 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "9eae0607571245ceba488d7aeaa520c1"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "a961c826e776689b3f78b13e381366a1"
server_key_iv: "495ca9cd313f277834ca8c9242c5d694"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.186930 21446 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:37.195097 21446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:37.200644 21446 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:37.200771 21446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:37.207314 21446 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39553
I20250903 14:26:37.207329 21523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:39553 every 8 connection(s)
I20250903 14:26:37.208119 21446 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:37.210408 21524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:37.214509 21524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Bootstrap starting.
I20250903 14:26:37.216342 21524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:37.216935 21524 log.cc:826] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:37.217589 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21446
I20250903 14:26:37.217682 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:37.217857 19718 external_mini_cluster.cc:1468] Setting key 834be20ccd5c42b115529b1412394c8b
I20250903 14:26:37.218430 21524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: No bootstrap required, opened a new log
I20250903 14:26:37.220857 21524 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } }
I20250903 14:26:37.221036 21524 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:37.221084 21524 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eae0607571245ceba488d7aeaa520c1, State: Initialized, Role: FOLLOWER
I20250903 14:26:37.221426 21524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } }
I20250903 14:26:37.221560 21524 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:37.221618 21524 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:37.221714 21524 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:37.222456 21524 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } }
I20250903 14:26:37.222793 21524 leader_election.cc:304] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eae0607571245ceba488d7aeaa520c1; no voters:
I20250903 14:26:37.223053 21524 leader_election.cc:290] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:37.223142 21529 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:37.223354 21529 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [term 1 LEADER]: Becoming Leader. State: Replica: 9eae0607571245ceba488d7aeaa520c1, State: Running, Role: LEADER
I20250903 14:26:37.223753 21529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } }
I20250903 14:26:37.224149 21524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:37.224828 21530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eae0607571245ceba488d7aeaa520c1. Latest consensus state: current_term: 1 leader_uuid: "9eae0607571245ceba488d7aeaa520c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } } }
I20250903 14:26:37.224922 21530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:37.225425 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eae0607571245ceba488d7aeaa520c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae0607571245ceba488d7aeaa520c1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39553 } } }
I20250903 14:26:37.225543 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:37.225803 21537 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:37.228369 21537 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:37.229869 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.218691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.19.65.190:39553)
0903 14:26:37.219057 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.219065 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.219091 (+ 26us) server_negotiation.cc:408] Connection header received
0903 14:26:37.219849 (+ 758us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.219864 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.220087 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.220293 (+ 206us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.221189 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.222002 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.222776 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.223294 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.225572 (+ 2278us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.225593 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.225606 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.225645 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.227595 (+ 1950us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.228010 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.228015 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.228019 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.228093 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.228311 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.228314 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.228316 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.228653 (+ 337us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.228829 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.229063 (+ 234us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.229259 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20250903 14:26:37.232656 21537 catalog_manager.cc:1349] Generated new cluster ID: ba5d891f4bc0475eb890a559800c6fd8
I20250903 14:26:37.232707 21537 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:37.240680 21537 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:37.241636 21537 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:37.250980 21537 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Generated new TSK 0
I20250903 14:26:37.251577 21537 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:37.290498 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39553
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:37.381112 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:37.381323 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:37.381402 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:37.384331 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:37.384382 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:37.384454 21552 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 14:26:37.384485 21552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:37.384536 21552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:37.387893 21552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:37.388644 21552 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:37.389303 21552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:37.394516 21558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.394585 21560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.394542 21557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:37.394696 21552 server_base.cc:1047] running on GCE node
I20250903 14:26:37.395047 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:37.395462 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:37.396608 21552 hybrid_clock.cc:648] HybridClock initialized: now 1756909597396595 us; error 25 us; skew 500 ppm
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:37.398838 21552 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:37.399626 21552 webserver.cc:480] Webserver started at http://127.19.65.129:46137/ using document root <none> and password file <none>
I20250903 14:26:37.400061 21552 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:37.400106 21552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:37.400272 21552 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:37.401770 21552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "e16c81c2c821459488c8caddcbcb5ec7"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "8d4002c8464870b0a861232c395f3255"
server_key_iv: "4b154fa18b93bfa91a88ac1116d5fc93"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.402155 21552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "e16c81c2c821459488c8caddcbcb5ec7"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "8d4002c8464870b0a861232c395f3255"
server_key_iv: "4b154fa18b93bfa91a88ac1116d5fc93"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.404922 21552 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250903 14:26:37.406683 21567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.407513 21552 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:37.407611 21552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "e16c81c2c821459488c8caddcbcb5ec7"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "8d4002c8464870b0a861232c395f3255"
server_key_iv: "4b154fa18b93bfa91a88ac1116d5fc93"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.407683 21552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:37.417095 21552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:37.419072 21552 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:37.419198 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:37.420241 21552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:37.420286 21552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.420331 21552 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:37.420364 21552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:37.429970 21552 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:46021
I20250903 14:26:37.430269 21682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:46021 every 8 connection(s)
I20250903 14:26:37.430820 21552 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:26:37.430855 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.420749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:43153 (local address 127.19.65.190:39553)
0903 14:26:37.420855 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.420858 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.421457 (+ 599us) server_negotiation.cc:408] Connection header received
0903 14:26:37.422140 (+ 683us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.422143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.422199 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.422263 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.423591 (+ 1328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.424409 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.425253 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.425466 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.427523 (+ 2057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.427544 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.427548 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.427578 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.429175 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.429582 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.429585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.429587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.429640 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.429961 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.429964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.429965 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.430116 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.430189 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.430651 (+ 462us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.430744 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 14:26:37.431432 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.420995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.129:43153)
0903 14:26:37.421350 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.421377 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.421976 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.422397 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.422404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.422771 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.423423 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.423437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.424533 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.424539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.425109 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.425117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.425271 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.425834 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.425859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.427374 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.429268 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.429273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.429279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.429458 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.429748 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.429754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.429756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.429854 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.430205 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.430210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.430414 (+ 204us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.430599 (+ 185us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.430822 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
I20250903 14:26:37.432508 21683 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39553
I20250903 14:26:37.432668 21683 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:37.433104 21683 heartbeater.cc:507] Master 127.19.65.190:39553 requested a full tablet report, sending...
I20250903 14:26:37.434418 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21552
I20250903 14:26:37.434525 21478 ts_manager.cc:194] Registered new tserver with Master: e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021)
I20250903 14:26:37.434582 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:37.434798 19718 external_mini_cluster.cc:1468] Setting key a76a28e26c625a9a824b09061375187f
I20250903 14:26:37.435810 21478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:43153
I20250903 14:26:37.441319 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.434413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:49657 (local address 127.19.65.190:39553)
0903 14:26:37.436295 (+ 1882us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.436298 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.436310 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:26:37.436337 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.436339 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.436382 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.436442 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.436914 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.437393 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.438226 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.438427 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.439040 (+ 613us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.439062 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.439064 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.439082 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.440197 (+ 1115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.440515 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.440518 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.440519 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.440553 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.440735 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.440738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.440739 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.440853 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.440926 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.441128 (+ 202us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.441215 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1811}
I20250903 14:26:37.441327 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.434338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.129:49657)
0903 14:26:37.434609 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.434625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.434704 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.436471 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.436474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.436619 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.436816 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.436821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.437538 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.437551 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.438109 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.438117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.438232 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.438718 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.438729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.438939 (+ 210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.440288 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.440291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.440293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.440437 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.440620 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.440623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.440624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.440666 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.440928 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.440932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.440997 (+ 65us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.441079 (+ 82us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.441184 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:37.472810 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39553
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:37.561285 21694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:37.561434 21694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:37.561535 21694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:37.564247 21694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:37.564296 21694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:37.564347 21694 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 14:26:37.564364 21694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:37.564401 21694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:37.567840 21694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:37.568573 21694 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:37.569200 21694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:37.574472 21700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.574472 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.574466 21699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:37.574525 21694 server_base.cc:1047] running on GCE node
I20250903 14:26:37.574955 21694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:37.575356 21694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:37.576488 21694 hybrid_clock.cc:648] HybridClock initialized: now 1756909597576479 us; error 27 us; skew 500 ppm
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:37.578723 21694 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:37.579582 21694 webserver.cc:480] Webserver started at http://127.19.65.130:44891/ using document root <none> and password file <none>
I20250903 14:26:37.580020 21694 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:37.580065 21694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:37.580207 21694 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:37.581596 21694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "533c1866188445b18694d58577c087ba"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "69b62efd641e54352cb61e32aa223fe6"
server_key_iv: "86fa6f43796a21d54294fafa96738068"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.582024 21694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "533c1866188445b18694d58577c087ba"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "69b62efd641e54352cb61e32aa223fe6"
server_key_iv: "86fa6f43796a21d54294fafa96738068"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.584775 21694 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:26:37.586505 21709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.587299 21694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:37.587386 21694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "533c1866188445b18694d58577c087ba"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "69b62efd641e54352cb61e32aa223fe6"
server_key_iv: "86fa6f43796a21d54294fafa96738068"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.587456 21694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:37.597463 21694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:37.599738 21694 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:37.599879 21694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:37.600950 21694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:37.600996 21694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.601042 21694 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:37.601071 21694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:37.610602 21694 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:42763
I20250903 14:26:37.610639 21824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:42763 every 8 connection(s)
I20250903 14:26:37.611266 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.601426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:35243 (local address 127.19.65.190:39553)
0903 14:26:37.601585 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.601591 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.602158 (+ 567us) server_negotiation.cc:408] Connection header received
0903 14:26:37.603112 (+ 954us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.603118 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.603183 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.603283 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.604858 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.605342 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.606133 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.606315 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.608351 (+ 2036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.608366 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.608369 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.608390 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.609725 (+ 1335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.610116 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.610119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.610121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.610164 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.610379 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.610382 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.610383 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.610537 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.610634 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.611032 (+ 398us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.611132 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20250903 14:26:37.611343 21694 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:26:37.611802 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.601650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.130:35243)
0903 14:26:37.602020 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.602064 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.602922 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.603459 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.603469 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.603999 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.604723 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.604737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.605436 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.605442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.606011 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.606023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.606209 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.606752 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.606786 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.608226 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.609830 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.609835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.609843 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.610033 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.610237 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.610242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.610244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.610306 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.610635 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.610641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.610836 (+ 195us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.610985 (+ 149us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.611190 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20250903 14:26:37.612958 21825 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39553
I20250903 14:26:37.613080 21825 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:37.613552 21825 heartbeater.cc:507] Master 127.19.65.190:39553 requested a full tablet report, sending...
I20250903 14:26:37.614415 21477 ts_manager.cc:194] Registered new tserver with Master: 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:37.615061 21477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:35243
I20250903 14:26:37.616750 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21694
I20250903 14:26:37.616860 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:37.617063 19718 external_mini_cluster.cc:1468] Setting key 439c04d74e347e1f069c3418800815cc
I20250903 14:26:37.619661 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.614790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:36499 (local address 127.19.65.190:39553)
0903 14:26:37.614893 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.614896 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.614991 (+ 95us) server_negotiation.cc:408] Connection header received
0903 14:26:37.615075 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.615077 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.615107 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.615177 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.615612 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.616002 (+ 390us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.616485 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.616658 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.617329 (+ 671us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.617344 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.617347 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.617364 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.618534 (+ 1170us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.618852 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.618854 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.618856 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.618888 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.619071 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.619074 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.619075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.619198 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.619274 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.619466 (+ 192us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.619554 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:26:37.619660 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.614701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.130:36499)
0903 14:26:37.614922 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.614934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.615014 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.615199 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.615202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.615339 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.615514 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.615520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.616092 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.616095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.616396 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.616402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.616491 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.616981 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.616992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.617235 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.618622 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.618625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.618627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.618772 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.618963 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.618965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.618967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.619003 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.619275 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.619278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.619341 (+ 63us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.619418 (+ 77us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.619518 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:37.653923 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39553
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:37.742338 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:37.742511 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:37.742599 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:37.745411 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:37.745460 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:37.745556 21836 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 14:26:37.745581 21836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:26:37.745640 21836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:37.748957 21836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:37.749786 21836 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:37.750463 21836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:37.755982 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.755990 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:37.755990 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:37.756191 21836 server_base.cc:1047] running on GCE node
I20250903 14:26:37.756481 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:37.756917 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:37.758049 21836 hybrid_clock.cc:648] HybridClock initialized: now 1756909597758036 us; error 29 us; skew 500 ppm
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:37.760237 21836 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:37.761021 21836 webserver.cc:480] Webserver started at http://127.19.65.131:38299/ using document root <none> and password file <none>
I20250903 14:26:37.761444 21836 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:37.761514 21836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:37.761683 21836 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:37.763060 21836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "ac33cb8d699d474493d7169da742f701"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "d3695e313d1967de4309765264965903"
server_key_iv: "a915ca4cdd5370bbb0570ae26e66ed77"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.763522 21836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "ac33cb8d699d474493d7169da742f701"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "d3695e313d1967de4309765264965903"
server_key_iv: "a915ca4cdd5370bbb0570ae26e66ed77"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.766294 21836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250903 14:26:37.768004 21851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.768802 21836 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:26:37.768875 21836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "ac33cb8d699d474493d7169da742f701"
format_stamp: "Formatted at 2025-09-03 14:26:37 on dist-test-slave-4wpb"
server_key: "d3695e313d1967de4309765264965903"
server_key_iv: "a915ca4cdd5370bbb0570ae26e66ed77"
server_key_version: "encryptionkey@0"
I20250903 14:26:37.768951 21836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:37.783733 21836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:37.786110 21836 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:37.786237 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:37.787283 21836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:37.787323 21836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:37.787358 21836 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:37.787385 21836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:37.797210 21836 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:42449
I20250903 14:26:37.797225 21966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:42449 every 8 connection(s)
I20250903 14:26:37.798045 21836 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:37.798374 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.787920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:44699 (local address 127.19.65.190:39553)
0903 14:26:37.788031 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.788034 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.788558 (+ 524us) server_negotiation.cc:408] Connection header received
0903 14:26:37.789233 (+ 675us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.789236 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.789292 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.789367 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.790892 (+ 1525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.791589 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.792604 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.792785 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.795072 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.795087 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.795089 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.795117 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.796727 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.797185 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.797187 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.797189 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.797226 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.797509 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.797515 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.797517 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.797646 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.797728 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.798137 (+ 409us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.798239 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250903 14:26:37.798897 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.788078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.131:44699)
0903 14:26:37.788435 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.788462 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.789061 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.789574 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.789584 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.790056 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.790757 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.790771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.791744 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.791774 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.792477 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.792489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.792680 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.793217 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.793271 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.794946 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.796831 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.796838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.796847 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.797093 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.797346 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.797348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.797350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.797416 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.797742 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.797749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.797930 (+ 181us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.798085 (+ 155us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.798297 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20250903 14:26:37.799806 21967 heartbeater.cc:344] Connected to a master server at 127.19.65.190:39553
I20250903 14:26:37.799932 21967 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:37.800354 21967 heartbeater.cc:507] Master 127.19.65.190:39553 requested a full tablet report, sending...
I20250903 14:26:37.801185 21478 ts_manager.cc:194] Registered new tserver with Master: ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449)
I20250903 14:26:37.801826 21478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:44699
I20250903 14:26:37.806479 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.801552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.19.65.131:49943)
0903 14:26:37.801759 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.801768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.801849 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.802091 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.802094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.802252 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:37.802455 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.802461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.803253 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.803255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.803614 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.803621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.803703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.804125 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:37.804136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:37.804366 (+ 230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:37.805715 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.805717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.805718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.805843 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.805985 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:37.805987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:37.805988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:37.806019 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:37.806203 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:37.806205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:37.806255 (+ 50us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.806312 (+ 57us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.806384 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20250903 14:26:37.806494 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.801583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:49943 (local address 127.19.65.190:39553)
0903 14:26:37.801684 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.801688 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.801824 (+ 136us) server_negotiation.cc:408] Connection header received
0903 14:26:37.801942 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.801945 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.801984 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.802057 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.802564 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.803137 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.803712 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.803841 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.804478 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.804489 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.804491 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.804513 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.805636 (+ 1123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.805903 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.805905 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.805906 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.805935 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.806060 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.806062 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.806063 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.806154 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.806208 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.806343 (+ 135us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.806407 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250903 14:26:37.807574 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 21836
I20250903 14:26:37.807677 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:37.807924 19718 external_mini_cluster.cc:1468] Setting key f943741b17334df469235c784ebc7329
I20250903 14:26:37.809412 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 14:26:37 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:37.828363 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.822448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.19.65.190:39553)
0903 14:26:37.822541 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.822544 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.822553 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:26:37.822578 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.822580 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.822610 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.822668 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:37.823294 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.823729 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.824282 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.824390 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.826165 (+ 1775us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:37.826177 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:37.826179 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:37.826198 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:37.827322 (+ 1124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.827696 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.827698 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.827699 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.827729 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:37.827865 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:37.827867 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:37.827868 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:37.827959 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:37.828039 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.828158 (+ 119us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.828225 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 14:26:37.830368 21478 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:37.832342 21478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:37.842473 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.838850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42449 (local address 127.0.0.1:55886)
0903 14:26:37.839414 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.839446 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.839567 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.840353 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.840359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.840380 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:37.840629 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.840638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.841465 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.841468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.842083 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.842090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.842186 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.842229 (+ 43us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.842299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.842366 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"spinlock_wait_cycles":1920,"thread_start_us":67,"threads_started":1}
I20250903 14:26:37.842896 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.839403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:42763 (local address 127.0.0.1:59562)
0903 14:26:37.840001 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.840013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.840084 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.840503 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.840506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.840522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:37.840715 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.840721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.841890 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.841893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.842604 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.842614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.842698 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.842713 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.842751 (+ 38us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.842791 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":82,"threads_started":1}
I20250903 14:26:37.843082 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.838957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46021 (local address 127.0.0.1:46034)
0903 14:26:37.839745 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:37.839756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:37.839833 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:37.840376 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:37.840378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:37.840389 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:37.840859 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.840866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.842034 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.842037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:37.842723 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:37.842730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.842837 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.842851 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 14:26:37.842889 (+ 38us) client_negotiation.cc:240] Negotiation successful
0903 14:26:37.842944 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":88,"threads_started":1}
I20250903 14:26:37.843719 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.839329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.19.65.129:46021)
0903 14:26:37.839647 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.839651 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.839816 (+ 165us) server_negotiation.cc:408] Connection header received
0903 14:26:37.839952 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.839958 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.840116 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.840235 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:37.840962 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.841945 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.842860 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.843342 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.843443 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.843503 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.843583 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20250903 14:26:37.843899 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.839540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.19.65.130:42763)
0903 14:26:37.839837 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.839842 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.840161 (+ 319us) server_negotiation.cc:408] Connection header received
0903 14:26:37.840242 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.840247 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.840386 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.840500 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:37.840814 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.841788 (+ 974us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.843029 (+ 1241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.843530 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.843618 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.843673 (+ 55us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.843758 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
I20250903 14:26:37.844050 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:37.838950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.19.65.131:42449)
0903 14:26:37.840030 (+ 1080us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:37.840035 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:37.840052 (+ 17us) server_negotiation.cc:408] Connection header received
0903 14:26:37.840106 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:37.840110 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:37.840221 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:37.840540 (+ 319us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:37.840729 (+ 189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.841379 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:37.842195 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:37.843690 (+ 1495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:37.843760 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:37.843817 (+ 57us) server_negotiation.cc:300] Negotiation successful
0903 14:26:37.843898 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":86,"threads_started":1}
I20250903 14:26:37.845955 21759 tablet_service.cc:1468] Processing CreateTablet for tablet a39856b5a86246cebb6fec86506f53f6 (DEFAULT_TABLE table=test-table [id=e3a74027c7d64e74a74421d6df91950d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:37.845918 21900 tablet_service.cc:1468] Processing CreateTablet for tablet a39856b5a86246cebb6fec86506f53f6 (DEFAULT_TABLE table=test-table [id=e3a74027c7d64e74a74421d6df91950d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:37.845918 21616 tablet_service.cc:1468] Processing CreateTablet for tablet a39856b5a86246cebb6fec86506f53f6 (DEFAULT_TABLE table=test-table [id=e3a74027c7d64e74a74421d6df91950d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:37.846655 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a39856b5a86246cebb6fec86506f53f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:37.846665 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a39856b5a86246cebb6fec86506f53f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:37.846665 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a39856b5a86246cebb6fec86506f53f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:37.851112 21990 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Bootstrap starting.
I20250903 14:26:37.852198 21991 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Bootstrap starting.
I20250903 14:26:37.852870 21990 tablet_bootstrap.cc:654] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:37.853926 21990 log.cc:826] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:37.854043 21992 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Bootstrap starting.
I20250903 14:26:37.854375 21991 tablet_bootstrap.cc:654] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:37.855371 21991 log.cc:826] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:37.855562 21992 tablet_bootstrap.cc:654] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:37.855603 21990 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: No bootstrap required, opened a new log
I20250903 14:26:37.855769 21990 ts_tablet_manager.cc:1397] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:26:37.856462 21992 log.cc:826] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:37.856894 21991 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: No bootstrap required, opened a new log
I20250903 14:26:37.857029 21991 ts_tablet_manager.cc:1397] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:26:37.857915 21990 raft_consensus.cc:357] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.858016 21992 tablet_bootstrap.cc:492] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: No bootstrap required, opened a new log
I20250903 14:26:37.858089 21990 raft_consensus.cc:383] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:37.858155 21990 raft_consensus.cc:738] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c1866188445b18694d58577c087ba, State: Initialized, Role: FOLLOWER
I20250903 14:26:37.858172 21992 ts_tablet_manager.cc:1397] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:26:37.858551 21990 consensus_queue.cc:260] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.859158 21825 heartbeater.cc:499] Master 127.19.65.190:39553 was elected leader, sending a full tablet report...
I20250903 14:26:37.859290 21990 ts_tablet_manager.cc:1428] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250903 14:26:37.859632 21991 raft_consensus.cc:357] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.859802 21991 raft_consensus.cc:383] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:37.859859 21991 raft_consensus.cc:738] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16c81c2c821459488c8caddcbcb5ec7, State: Initialized, Role: FOLLOWER
I20250903 14:26:37.860198 21991 consensus_queue.cc:260] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.860709 21683 heartbeater.cc:499] Master 127.19.65.190:39553 was elected leader, sending a full tablet report...
I20250903 14:26:37.860894 21991 ts_tablet_manager.cc:1428] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250903 14:26:37.862248 21826 tablet.cc:2378] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:26:37.863119 21992 raft_consensus.cc:357] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.863324 21992 raft_consensus.cc:383] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:37.863389 21992 raft_consensus.cc:738] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33cb8d699d474493d7169da742f701, State: Initialized, Role: FOLLOWER
I20250903 14:26:37.863706 21992 consensus_queue.cc:260] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:37.864245 21967 heartbeater.cc:499] Master 127.19.65.190:39553 was elected leader, sending a full tablet report...
I20250903 14:26:37.864385 21992 ts_tablet_manager.cc:1428] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250903 14:26:37.931794 21684 tablet.cc:2378] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:26:38.013942 21996 raft_consensus.cc:491] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:38.014139 21996 raft_consensus.cc:513] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:38.015194 21996 leader_election.cc:290] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449), e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021)
I20250903 14:26:38.017843 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.015351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42449 (local address 127.19.65.130:42679)
0903 14:26:38.015512 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.015527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.015624 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.015907 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.015909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.015932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:38.016178 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.016184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.016889 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.016893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.017424 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.017431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.017571 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.017595 (+ 24us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.017642 (+ 47us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.017693 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20250903 14:26:38.017875 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.015385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46021 (local address 127.19.65.130:35681)
0903 14:26:38.015777 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.015789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.015882 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.016128 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.016130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.016143 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:38.016383 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.016390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.017020 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.017023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.017497 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.017504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.017605 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.017621 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.017664 (+ 43us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.017710 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":141,"threads_started":1}
I20250903 14:26:38.018221 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.015545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:42679 (local address 127.19.65.131:42449)
0903 14:26:38.015683 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.015688 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.015704 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:26:38.015754 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.015757 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.015813 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.015888 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:38.016292 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.016794 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.017576 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.017984 (+ 408us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.018024 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.018069 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.018129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 14:26:38.018268 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.015555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:35681 (local address 127.19.65.129:46021)
0903 14:26:38.015692 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.015708 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.015853 (+ 145us) server_negotiation.cc:408] Connection header received
0903 14:26:38.015982 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.015985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.016045 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.016110 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:38.016482 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.016924 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.017611 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.018054 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.018085 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.018126 (+ 41us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.018170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 14:26:38.018714 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a39856b5a86246cebb6fec86506f53f6" candidate_uuid: "533c1866188445b18694d58577c087ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7" is_pre_election: true
I20250903 14:26:38.018715 21919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a39856b5a86246cebb6fec86506f53f6" candidate_uuid: "533c1866188445b18694d58577c087ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac33cb8d699d474493d7169da742f701" is_pre_election: true
I20250903 14:26:38.018918 21919 raft_consensus.cc:2466] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533c1866188445b18694d58577c087ba in term 0.
I20250903 14:26:38.018932 21637 raft_consensus.cc:2466] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533c1866188445b18694d58577c087ba in term 0.
I20250903 14:26:38.019272 21710 leader_election.cc:304] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533c1866188445b18694d58577c087ba, ac33cb8d699d474493d7169da742f701; no voters:
I20250903 14:26:38.019480 21996 raft_consensus.cc:2802] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:38.019537 21996 raft_consensus.cc:491] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:38.019567 21996 raft_consensus.cc:3058] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.020363 21996 raft_consensus.cc:513] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:38.020637 21996 leader_election.cc:290] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [CANDIDATE]: Term 1 election: Requested vote from peers ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449), e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021)
I20250903 14:26:38.020915 21919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a39856b5a86246cebb6fec86506f53f6" candidate_uuid: "533c1866188445b18694d58577c087ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac33cb8d699d474493d7169da742f701"
I20250903 14:26:38.020915 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a39856b5a86246cebb6fec86506f53f6" candidate_uuid: "533c1866188445b18694d58577c087ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7"
I20250903 14:26:38.021014 21637 raft_consensus.cc:3058] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.021014 21919 raft_consensus.cc:3058] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.022087 21637 raft_consensus.cc:2466] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533c1866188445b18694d58577c087ba in term 1.
I20250903 14:26:38.022087 21919 raft_consensus.cc:2466] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533c1866188445b18694d58577c087ba in term 1.
I20250903 14:26:38.022365 21711 leader_election.cc:304] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533c1866188445b18694d58577c087ba, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:38.022496 21996 raft_consensus.cc:2802] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:38.022689 21996 raft_consensus.cc:695] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 1 LEADER]: Becoming Leader. State: Replica: 533c1866188445b18694d58577c087ba, State: Running, Role: LEADER
I20250903 14:26:38.022990 21996 consensus_queue.cc:237] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } }
I20250903 14:26:38.025691 21478 catalog_manager.cc:5582] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 533c1866188445b18694d58577c087ba (127.19.65.130). New cstate: current_term: 1 leader_uuid: "533c1866188445b18694d58577c087ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } health_report { overall_health: HEALTHY } } }
W20250903 14:26:38.049279 21968 tablet.cc:2378] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 03 14:26:38 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:38.092965 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.086883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39553 (local address 127.0.0.1:52570)
0903 14:26:38.087089 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.087100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.087180 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.087363 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.087365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.087713 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:38.087883 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.087888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.088427 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.088428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.088742 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.088746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.088815 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.089270 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:38.089280 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:38.090581 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:38.092023 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:38.092028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:38.092031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:38.092200 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:38.092349 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:38.092350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:38.092351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:38.092384 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:38.092594 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:38.092598 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:38.092671 (+ 73us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.092752 (+ 81us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.092840 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20250903 14:26:38.092985 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.086951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.19.65.190:39553)
0903 14:26:38.087079 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.087085 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.087155 (+ 70us) server_negotiation.cc:408] Connection header received
0903 14:26:38.087239 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.087241 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.087275 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.087332 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:38.087963 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.088359 (+ 396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.088811 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.088911 (+ 100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.090717 (+ 1806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:38.090730 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:38.090732 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:38.090751 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:38.091950 (+ 1199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:38.092264 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:38.092266 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:38.092267 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:38.092296 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:38.092432 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:38.092433 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:38.092434 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:38.092533 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:38.092593 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.092786 (+ 193us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.092885 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":58}
I20250903 14:26:38.096071 21478 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 14:26:38.096648 21478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:38.100479 21759 tablet_service.cc:1468] Processing CreateTablet for tablet c22c9144f7ae4de1beb0ca89c3e50531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51677276667a443aa3251e9b4acf6b50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:38.100688 21616 tablet_service.cc:1468] Processing CreateTablet for tablet c22c9144f7ae4de1beb0ca89c3e50531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51677276667a443aa3251e9b4acf6b50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:38.101068 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22c9144f7ae4de1beb0ca89c3e50531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.100911 21900 tablet_service.cc:1468] Processing CreateTablet for tablet c22c9144f7ae4de1beb0ca89c3e50531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51677276667a443aa3251e9b4acf6b50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:26:38.101226 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22c9144f7ae4de1beb0ca89c3e50531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.101351 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22c9144f7ae4de1beb0ca89c3e50531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.103039 21990 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: Bootstrap starting.
I20250903 14:26:38.103590 21992 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: Bootstrap starting.
I20250903 14:26:38.103988 21990 tablet_bootstrap.cc:654] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.104387 21992 tablet_bootstrap.cc:654] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.105193 21990 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: No bootstrap required, opened a new log
I20250903 14:26:38.105278 21990 ts_tablet_manager.cc:1397] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:38.105293 21991 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: Bootstrap starting.
I20250903 14:26:38.105742 21990 raft_consensus.cc:357] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.105837 21990 raft_consensus.cc:383] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.105854 21992 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: No bootstrap required, opened a new log
I20250903 14:26:38.105867 21990 raft_consensus.cc:738] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c1866188445b18694d58577c087ba, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.105918 21992 ts_tablet_manager.cc:1397] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:38.105968 21990 consensus_queue.cc:260] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.106048 21991 tablet_bootstrap.cc:654] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.106200 21990 ts_tablet_manager.cc:1428] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.106220 21996 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.106325 21996 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.106302 21992 raft_consensus.cc:357] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.106401 21992 raft_consensus.cc:383] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.106427 21992 raft_consensus.cc:738] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33cb8d699d474493d7169da742f701, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.106559 21992 consensus_queue.cc:260] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.106837 21992 ts_tablet_manager.cc:1428] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.107244 21991 tablet_bootstrap.cc:492] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: No bootstrap required, opened a new log
I20250903 14:26:38.107252 21998 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.107316 21991 ts_tablet_manager.cc:1397] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:38.107383 21998 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.107683 21991 raft_consensus.cc:357] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.107779 21991 raft_consensus.cc:383] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.107810 21991 raft_consensus.cc:738] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16c81c2c821459488c8caddcbcb5ec7, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.107913 21991 consensus_queue.cc:260] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.108134 21991 ts_tablet_manager.cc:1428] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.108106 21997 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.108191 21997 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.229439 21998 raft_consensus.cc:491] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:38.229559 21998 raft_consensus.cc:513] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.230460 21998 leader_election.cc:290] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:38.232776 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.230655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46021 (local address 127.19.65.131:48123)
0903 14:26:38.230775 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.230791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.230862 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.231122 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.231125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.231144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:38.231331 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.231336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.231947 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.231951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.232437 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.232443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.232537 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.232558 (+ 21us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.232603 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.232659 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 14:26:38.233068 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.230862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:42763 (local address 127.19.65.131:34463)
0903 14:26:38.231191 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.231200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.231278 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.231476 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.231478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.231492 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:26:38.231691 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.231696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.232271 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.232276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.232750 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.232756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.232847 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.232861 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.232904 (+ 43us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.232946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20250903 14:26:38.233215 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.230727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48123 (local address 127.19.65.129:46021)
0903 14:26:38.230844 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.230849 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.230863 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:26:38.230970 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.230974 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.231026 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.231097 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:38.231436 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.231854 (+ 418us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.232536 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.232933 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.232974 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.233026 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.233069 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:26:38.233569 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.230951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:34463 (local address 127.19.65.130:42763)
0903 14:26:38.231087 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.231093 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.231263 (+ 170us) server_negotiation.cc:408] Connection header received
0903 14:26:38.231358 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.231361 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.231397 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.231453 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:26:38.231793 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.232180 (+ 387us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.232855 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.233298 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.233341 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.233391 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.233434 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 14:26:38.233645 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22c9144f7ae4de1beb0ca89c3e50531" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7" is_pre_election: true
I20250903 14:26:38.233796 21637 raft_consensus.cc:2466] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:38.234055 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22c9144f7ae4de1beb0ca89c3e50531" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba" is_pre_election: true
I20250903 14:26:38.234113 21853 leader_election.cc:304] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac33cb8d699d474493d7169da742f701, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:38.234253 21779 raft_consensus.cc:2466] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:38.234306 21998 raft_consensus.cc:2802] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:38.234355 21998 raft_consensus.cc:491] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:38.234424 21998 raft_consensus.cc:3058] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.235204 21998 raft_consensus.cc:513] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.235494 21998 leader_election.cc:290] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Requested vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:38.235759 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22c9144f7ae4de1beb0ca89c3e50531" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7"
I20250903 14:26:38.235831 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22c9144f7ae4de1beb0ca89c3e50531" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba"
I20250903 14:26:38.235867 21637 raft_consensus.cc:3058] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.235925 21779 raft_consensus.cc:3058] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.236814 21637 raft_consensus.cc:2466] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:38.236814 21779 raft_consensus.cc:2466] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:38.237048 21853 leader_election.cc:304] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533c1866188445b18694d58577c087ba, ac33cb8d699d474493d7169da742f701; no voters:
I20250903 14:26:38.237172 21998 raft_consensus.cc:2802] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:38.237326 21998 raft_consensus.cc:695] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Becoming Leader. State: Replica: ac33cb8d699d474493d7169da742f701, State: Running, Role: LEADER
I20250903 14:26:38.237617 21998 consensus_queue.cc:237] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.239223 22012 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: TxnStatusTablet state changed. Reason: New leader ac33cb8d699d474493d7169da742f701. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.239288 22012 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:38.239490 22014 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 14:26:38.239639 22014 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 14:26:38.240270 21477 catalog_manager.cc:5582] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac33cb8d699d474493d7169da742f701 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } health_report { overall_health: HEALTHY } } }
I20250903 14:26:38.259312 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.256218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.19.65.131:42449)
0903 14:26:38.256309 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.256312 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.256322 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:26:38.256501 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.256503 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.256545 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.256611 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.256967 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.257791 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.258554 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.258659 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.258706 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.259032 (+ 326us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.259104 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.259198 (+ 94us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.259235 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250903 14:26:38.259361 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.256127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42449 (local address 127.0.0.1:55896)
0903 14:26:38.256275 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.256288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.256379 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.256627 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.256641 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.256649 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:38.256857 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.256863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.257899 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.257903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.258448 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.258470 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.258568 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.258606 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.259116 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.259128 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.259189 (+ 61us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.259251 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20250903 14:26:38.263064 21637 raft_consensus.cc:1273] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.263080 21779 raft_consensus.cc:1273] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.263583 22012 consensus_queue.cc:1035] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:38.263722 21998 consensus_queue.cc:1035] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:38.266238 21996 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: TxnStatusTablet state changed. Reason: New leader ac33cb8d699d474493d7169da742f701. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.266320 21996 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.266691 21997 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: TxnStatusTablet state changed. Reason: New leader ac33cb8d699d474493d7169da742f701. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.266791 21997 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.268880 22017 mvcc.cc:204] Tried to move back new op lower bound from 7196301714480803840 to 7196301714383446016. Current Snapshot: MvccSnapshot[applied={T|T < 7196301714480803840}]
I20250903 14:26:38.268985 22012 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.269070 22012 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:38.269361 21996 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.269454 21996 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P 533c1866188445b18694d58577c087ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.270941 21997 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.271019 21997 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P e16c81c2c821459488c8caddcbcb5ec7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:26:38.271286 22018 mvcc.cc:204] Tried to move back new op lower bound from 7196301714480803840 to 7196301714383446016. Current Snapshot: MvccSnapshot[applied={T|T < 7196301714480803840}]
I20250903 14:26:38.271871 21998 tablet_replica.cc:440] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } } }
I20250903 14:26:38.271951 21998 tablet_replica.cc:443] T c22c9144f7ae4de1beb0ca89c3e50531 P ac33cb8d699d474493d7169da742f701: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:26:38.280038 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.277325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.19.65.130:42763)
0903 14:26:38.277439 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.277443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.277461 (+ 18us) server_negotiation.cc:408] Connection header received
0903 14:26:38.277554 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.277557 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.277607 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.277682 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.277953 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.278578 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.279144 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.279307 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.279354 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.279756 (+ 402us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.279850 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.279896 (+ 46us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.279940 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 14:26:38.288012 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.285766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:48473 (local address 127.19.65.131:42449)
0903 14:26:38.285895 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.285898 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.285963 (+ 65us) server_negotiation.cc:408] Connection header received
0903 14:26:38.286055 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.286057 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.286089 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.286142 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.286501 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.286871 (+ 370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.287603 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.287701 (+ 98us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.287730 (+ 29us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.287778 (+ 48us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.287823 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.287911 (+ 88us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.287941 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250903 14:26:38.288076 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.285651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42449 (local address 127.19.65.130:48473)
0903 14:26:38.285893 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.285905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.285990 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.286162 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.286165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.286171 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:38.286415 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.286421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.286956 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.286959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.287513 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.287521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.287612 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.287640 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.287845 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.287849 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.287905 (+ 56us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.287961 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20250903 14:26:38.294595 21919 raft_consensus.cc:1273] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Refusing update from remote peer 533c1866188445b18694d58577c087ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.294737 21637 raft_consensus.cc:1273] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer 533c1866188445b18694d58577c087ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.295080 21996 consensus_queue.cc:1035] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:38.295209 22000 consensus_queue.cc:1035] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:38.298132 22020 mvcc.cc:204] Tried to move back new op lower bound from 7196301714609844224 to 7196301713504575488. Current Snapshot: MvccSnapshot[applied={T|T < 7196301714609844224}]
I20250903 14:26:38.313962 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.311113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46063 (local address 127.19.65.130:42763)
0903 14:26:38.311209 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.311213 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.311224 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:26:38.311315 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.311318 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.311358 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.311420 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.311855 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.312225 (+ 370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.313075 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.313265 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.313536 (+ 271us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.313612 (+ 76us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.313673 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.313843 (+ 170us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.313881 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20250903 14:26:38.314025 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.310864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:42763 (local address 127.19.65.131:46063)
0903 14:26:38.311095 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.311107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.311187 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.311508 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.311511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.311518 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:38.311765 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.311771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.312301 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.312304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.312946 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.312955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.313332 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.313366 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.313748 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.313753 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.313820 (+ 67us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.313883 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":58,"threads_started":1}
I20250903 14:26:38.334606 21477 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:38.334759 21477 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:38.337076 21477 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Sending DeleteTablet for 3 replicas of tablet a39856b5a86246cebb6fec86506f53f6
I20250903 14:26:38.337896 21616 tablet_service.cc:1515] Processing DeleteTablet for tablet a39856b5a86246cebb6fec86506f53f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46034
I20250903 14:26:38.338162 21759 tablet_service.cc:1515] Processing DeleteTablet for tablet a39856b5a86246cebb6fec86506f53f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59562
I20250903 14:26:38.338423 22037 tablet_replica.cc:331] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: stopping tablet replica
I20250903 14:26:38.338644 22037 raft_consensus.cc:2241] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:38.338886 22037 raft_consensus.cc:2270] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:38.339144 21477 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:38.339522 21477 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:38.340201 21900 tablet_service.cc:1515] Processing DeleteTablet for tablet a39856b5a86246cebb6fec86506f53f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55886
I20250903 14:26:38.340852 22038 tablet_replica.cc:331] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: stopping tablet replica
I20250903 14:26:38.341037 22039 tablet_replica.cc:331] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: stopping tablet replica
I20250903 14:26:38.341256 22038 raft_consensus.cc:2241] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:38.341497 22039 raft_consensus.cc:2241] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:38.342499 22038 raft_consensus.cc:2270] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:38.342669 22039 raft_consensus.cc:2270] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:38.344400 21900 tablet_service.cc:1468] Processing CreateTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 (DEFAULT_TABLE table=test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:38.344679 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be9597b75f894e4bb9a6bb901dc48cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.344638 21616 tablet_service.cc:1468] Processing CreateTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 (DEFAULT_TABLE table=test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:38.344868 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be9597b75f894e4bb9a6bb901dc48cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.346714 21992 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Bootstrap starting.
I20250903 14:26:38.347002 21759 tablet_service.cc:1468] Processing CreateTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 (DEFAULT_TABLE table=test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:38.347214 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be9597b75f894e4bb9a6bb901dc48cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:38.347496 21992 tablet_bootstrap.cc:654] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.348816 22037 ts_tablet_manager.cc:1905] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:38.349193 21991 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Bootstrap starting.
I20250903 14:26:38.350180 21991 tablet_bootstrap.cc:654] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.350688 21990 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Bootstrap starting.
I20250903 14:26:38.351138 22037 ts_tablet_manager.cc:1918] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:38.351202 22037 log.cc:1199] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/a39856b5a86246cebb6fec86506f53f6
I20250903 14:26:38.351454 22037 ts_tablet_manager.cc:1939] T a39856b5a86246cebb6fec86506f53f6 P e16c81c2c821459488c8caddcbcb5ec7: Deleting consensus metadata
I20250903 14:26:38.351497 21990 tablet_bootstrap.cc:654] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:38.352583 21463 catalog_manager.cc:4928] TS e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021): tablet a39856b5a86246cebb6fec86506f53f6 (table test-table [id=e3a74027c7d64e74a74421d6df91950d]) successfully deleted
I20250903 14:26:38.353199 22039 ts_tablet_manager.cc:1905] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:38.354110 21991 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: No bootstrap required, opened a new log
I20250903 14:26:38.354199 21991 ts_tablet_manager.cc:1397] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250903 14:26:38.354615 21991 raft_consensus.cc:357] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.354696 21991 raft_consensus.cc:383] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.354720 21991 raft_consensus.cc:738] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16c81c2c821459488c8caddcbcb5ec7, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.354815 21991 consensus_queue.cc:260] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.355002 21991 ts_tablet_manager.cc:1428] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.357021 22038 ts_tablet_manager.cc:1905] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:38.359127 22038 ts_tablet_manager.cc:1918] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:38.359205 22038 log.cc:1199] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/a39856b5a86246cebb6fec86506f53f6
I20250903 14:26:38.359757 22039 ts_tablet_manager.cc:1918] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:38.359839 21990 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: No bootstrap required, opened a new log
I20250903 14:26:38.359912 21990 ts_tablet_manager.cc:1397] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250903 14:26:38.360072 22038 ts_tablet_manager.cc:1939] T a39856b5a86246cebb6fec86506f53f6 P 533c1866188445b18694d58577c087ba: Deleting consensus metadata
I20250903 14:26:38.360302 21990 raft_consensus.cc:357] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.360419 21990 raft_consensus.cc:383] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.360448 21990 raft_consensus.cc:738] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c1866188445b18694d58577c087ba, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.360549 21990 consensus_queue.cc:260] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.360584 22039 log.cc:1199] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/a39856b5a86246cebb6fec86506f53f6
I20250903 14:26:38.360743 21990 ts_tablet_manager.cc:1428] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.360822 22039 ts_tablet_manager.cc:1939] T a39856b5a86246cebb6fec86506f53f6 P ac33cb8d699d474493d7169da742f701: Deleting consensus metadata
I20250903 14:26:38.362381 21463 catalog_manager.cc:4928] TS 533c1866188445b18694d58577c087ba (127.19.65.130:42763): tablet a39856b5a86246cebb6fec86506f53f6 (table test-table [id=e3a74027c7d64e74a74421d6df91950d]) successfully deleted
I20250903 14:26:38.362447 21462 catalog_manager.cc:4928] TS ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449): tablet a39856b5a86246cebb6fec86506f53f6 (table test-table [id=e3a74027c7d64e74a74421d6df91950d]) successfully deleted
I20250903 14:26:38.363153 21992 tablet_bootstrap.cc:492] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: No bootstrap required, opened a new log
I20250903 14:26:38.363261 21992 ts_tablet_manager.cc:1397] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250903 14:26:38.363652 21992 raft_consensus.cc:357] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.363746 21992 raft_consensus.cc:383] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:38.363776 21992 raft_consensus.cc:738] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33cb8d699d474493d7169da742f701, State: Initialized, Role: FOLLOWER
I20250903 14:26:38.363879 21992 consensus_queue.cc:260] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.364100 21992 ts_tablet_manager.cc:1428] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:38.478955 22012 raft_consensus.cc:491] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:38.479080 22012 raft_consensus.cc:513] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.479476 22012 leader_election.cc:290] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:38.479847 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be9597b75f894e4bb9a6bb901dc48cd8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7" is_pre_election: true
I20250903 14:26:38.479913 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be9597b75f894e4bb9a6bb901dc48cd8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba" is_pre_election: true
I20250903 14:26:38.479981 21637 raft_consensus.cc:2466] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:38.480050 21779 raft_consensus.cc:2466] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:38.480280 21853 leader_election.cc:304] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac33cb8d699d474493d7169da742f701, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:38.480492 22012 raft_consensus.cc:2802] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:38.480556 22012 raft_consensus.cc:491] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:38.480593 22012 raft_consensus.cc:3058] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.481748 22012 raft_consensus.cc:513] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.482059 22012 leader_election.cc:290] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Requested vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:38.482331 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be9597b75f894e4bb9a6bb901dc48cd8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7"
I20250903 14:26:38.482372 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be9597b75f894e4bb9a6bb901dc48cd8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba"
I20250903 14:26:38.482432 21637 raft_consensus.cc:3058] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.482481 21779 raft_consensus.cc:3058] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:38.483170 21637 raft_consensus.cc:2466] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:38.483462 21853 leader_election.cc:304] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac33cb8d699d474493d7169da742f701, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:38.483459 21779 raft_consensus.cc:2466] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:38.483768 22012 raft_consensus.cc:2802] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:38.483867 22012 raft_consensus.cc:695] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Becoming Leader. State: Replica: ac33cb8d699d474493d7169da742f701, State: Running, Role: LEADER
I20250903 14:26:38.484006 22012 consensus_queue.cc:237] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:38.485613 21477 catalog_manager.cc:5582] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac33cb8d699d474493d7169da742f701 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } health_report { overall_health: HEALTHY } } }
I20250903 14:26:38.509590 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.507391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.19.65.131:42449)
0903 14:26:38.507504 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.507508 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.507523 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:26:38.507627 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.507630 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.507693 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.507758 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.508017 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.508562 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.509045 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.509192 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.509231 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.509293 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.509355 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.509423 (+ 68us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.509466 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:26:38.516120 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.514301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46421 (local address 127.19.65.131:42449)
0903 14:26:38.514423 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:38.514427 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:38.514449 (+ 22us) server_negotiation.cc:408] Connection header received
0903 14:26:38.514530 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:38.514532 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:38.514565 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:38.514611 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:38.514825 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.515169 (+ 344us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.515680 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.515782 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.515810 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.515857 (+ 47us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.515907 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:38.516002 (+ 95us) server_negotiation.cc:300] Negotiation successful
0903 14:26:38.516031 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:26:38.516121 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:38.514239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42449 (local address 127.19.65.131:46421)
0903 14:26:38.514386 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:38.514398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:38.514463 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:38.514622 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:38.514624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:38.514630 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:26:38.514756 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.514760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.515235 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:38.515237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:38.515595 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:38.515600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:38.515673 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:38.515689 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:26:38.515926 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:26:38.515929 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:26:38.515980 (+ 51us) client_negotiation.cc:240] Negotiation successful
0903 14:26:38.516023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20250903 14:26:38.520447 21779 raft_consensus.cc:1273] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.520507 21637 raft_consensus.cc:1273] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:38.520771 21998 consensus_queue.cc:1035] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:38.520872 22025 consensus_queue.cc:1035] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 14:26:38.532058 22017 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250903 14:26:40.075776 21478 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:40.075907 21478 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52566:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:40.077790 21478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Sending DeleteTablet for 3 replicas of tablet be9597b75f894e4bb9a6bb901dc48cd8
I20250903 14:26:40.078275 21616 tablet_service.cc:1515] Processing DeleteTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46034
I20250903 14:26:40.078329 21759 tablet_service.cc:1515] Processing DeleteTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59562
I20250903 14:26:40.078554 21900 tablet_service.cc:1515] Processing DeleteTablet for tablet be9597b75f894e4bb9a6bb901dc48cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55886
I20250903 14:26:40.078624 22070 tablet_replica.cc:331] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: stopping tablet replica
I20250903 14:26:40.078855 22069 tablet_replica.cc:331] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: stopping tablet replica
I20250903 14:26:40.078864 22070 raft_consensus.cc:2241] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:40.079006 22069 raft_consensus.cc:2241] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:40.079042 22070 raft_consensus.cc:2270] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.079136 22069 raft_consensus.cc:2270] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.079185 22071 tablet_replica.cc:331] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: stopping tablet replica
I20250903 14:26:40.079473 22071 raft_consensus.cc:2241] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:40.079706 22070 ts_tablet_manager.cc:1905] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.079775 22071 raft_consensus.cc:2270] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.080116 22069 ts_tablet_manager.cc:1905] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.080276 22071 ts_tablet_manager.cc:1905] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.082337 22071 ts_tablet_manager.cc:1918] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 14:26:40.082456 22071 log.cc:1199] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/be9597b75f894e4bb9a6bb901dc48cd8
I20250903 14:26:40.082792 22071 ts_tablet_manager.cc:1939] T be9597b75f894e4bb9a6bb901dc48cd8 P ac33cb8d699d474493d7169da742f701: Deleting consensus metadata
I20250903 14:26:40.083384 21462 catalog_manager.cc:4928] TS ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449): tablet be9597b75f894e4bb9a6bb901dc48cd8 (table test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]) successfully deleted
I20250903 14:26:40.084525 22069 ts_tablet_manager.cc:1918] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 14:26:40.084581 22069 log.cc:1199] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/be9597b75f894e4bb9a6bb901dc48cd8
I20250903 14:26:40.084765 22069 ts_tablet_manager.cc:1939] T be9597b75f894e4bb9a6bb901dc48cd8 P e16c81c2c821459488c8caddcbcb5ec7: Deleting consensus metadata
I20250903 14:26:40.084908 22070 ts_tablet_manager.cc:1918] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 14:26:40.084978 22070 log.cc:1199] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/be9597b75f894e4bb9a6bb901dc48cd8
I20250903 14:26:40.085273 22070 ts_tablet_manager.cc:1939] T be9597b75f894e4bb9a6bb901dc48cd8 P 533c1866188445b18694d58577c087ba: Deleting consensus metadata
I20250903 14:26:40.085420 21463 catalog_manager.cc:4928] TS e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021): tablet be9597b75f894e4bb9a6bb901dc48cd8 (table test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]) successfully deleted
I20250903 14:26:40.085856 21463 catalog_manager.cc:4928] TS 533c1866188445b18694d58577c087ba (127.19.65.130:42763): tablet be9597b75f894e4bb9a6bb901dc48cd8 (table test-table [id=ecc0fcdd2d8e4ffaa74d0c333acf3dc2]) successfully deleted
Sep 03 14:26:40 dist-test-slave-4wpb krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 14:26:40 dist-test-slave-4wpb krb5kdc[21430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:40.100315 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:40.094137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.19.65.190:39553)
0903 14:26:40.094378 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:40.094381 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:40.094441 (+ 60us) server_negotiation.cc:408] Connection header received
0903 14:26:40.094530 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:40.094532 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:40.094567 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:40.094634 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:40.095239 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:40.095691 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:40.096194 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:40.096302 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:40.098052 (+ 1750us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:40.098065 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:40.098067 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:40.098086 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:40.099333 (+ 1247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:40.099606 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:40.099608 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:40.099609 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:40.099639 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:40.099759 (+ 120us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:40.099761 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:40.099762 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:40.099864 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:40.099943 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:40.100047 (+ 104us) server_negotiation.cc:300] Negotiation successful
0903 14:26:40.100138 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20250903 14:26:40.102211 21478 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:40.102547 21478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:40.106195 21616 tablet_service.cc:1468] Processing CreateTablet for tablet 41c67b85399448918af0a25fe17c7efe (DEFAULT_TABLE table=test-table [id=d7b2209c3383454b98c44b918e93f9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.106221 21759 tablet_service.cc:1468] Processing CreateTablet for tablet 41c67b85399448918af0a25fe17c7efe (DEFAULT_TABLE table=test-table [id=d7b2209c3383454b98c44b918e93f9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.106302 21900 tablet_service.cc:1468] Processing CreateTablet for tablet 41c67b85399448918af0a25fe17c7efe (DEFAULT_TABLE table=test-table [id=d7b2209c3383454b98c44b918e93f9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.106424 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c67b85399448918af0a25fe17c7efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.106456 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c67b85399448918af0a25fe17c7efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.106487 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c67b85399448918af0a25fe17c7efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.107806 22080 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Bootstrap starting.
I20250903 14:26:40.108292 22081 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Bootstrap starting.
I20250903 14:26:40.108546 22080 tablet_bootstrap.cc:654] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.109087 22081 tablet_bootstrap.cc:654] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.109555 22080 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: No bootstrap required, opened a new log
I20250903 14:26:40.109628 22080 ts_tablet_manager.cc:1397] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:40.109632 22082 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Bootstrap starting.
I20250903 14:26:40.110001 22080 raft_consensus.cc:357] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.110070 22081 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: No bootstrap required, opened a new log
I20250903 14:26:40.110100 22080 raft_consensus.cc:383] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.110127 22080 raft_consensus.cc:738] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16c81c2c821459488c8caddcbcb5ec7, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.110133 22081 ts_tablet_manager.cc:1397] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250903 14:26:40.110320 22082 tablet_bootstrap.cc:654] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.110283 22080 consensus_queue.cc:260] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.110514 22081 raft_consensus.cc:357] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.110591 22080 ts_tablet_manager.cc:1428] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:40.110659 22081 raft_consensus.cc:383] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.110725 22081 raft_consensus.cc:738] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c1866188445b18694d58577c087ba, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.110845 22081 consensus_queue.cc:260] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.111342 22081 ts_tablet_manager.cc:1428] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:26:40.111951 22082 tablet_bootstrap.cc:492] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: No bootstrap required, opened a new log
I20250903 14:26:40.112033 22082 ts_tablet_manager.cc:1397] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:40.112435 22082 raft_consensus.cc:357] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.112517 22082 raft_consensus.cc:383] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.112547 22082 raft_consensus.cc:738] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33cb8d699d474493d7169da742f701, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.112675 22082 consensus_queue.cc:260] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.112924 22082 ts_tablet_manager.cc:1428] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:40.321545 22062 raft_consensus.cc:491] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:40.321655 22062 raft_consensus.cc:513] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.321979 22062 leader_election.cc:290] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:40.322474 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41c67b85399448918af0a25fe17c7efe" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7" is_pre_election: true
I20250903 14:26:40.322491 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41c67b85399448918af0a25fe17c7efe" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba" is_pre_election: true
I20250903 14:26:40.322606 21779 raft_consensus.cc:2466] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:40.322639 21637 raft_consensus.cc:2466] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:40.322901 21853 leader_election.cc:304] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533c1866188445b18694d58577c087ba, ac33cb8d699d474493d7169da742f701; no voters:
I20250903 14:26:40.323096 22062 raft_consensus.cc:2802] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:40.323172 22062 raft_consensus.cc:491] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:40.323225 22062 raft_consensus.cc:3058] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.324127 22062 raft_consensus.cc:513] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.324482 22062 leader_election.cc:290] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Requested vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:40.324797 21637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41c67b85399448918af0a25fe17c7efe" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7"
I20250903 14:26:40.324852 21779 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41c67b85399448918af0a25fe17c7efe" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba"
I20250903 14:26:40.324908 21637 raft_consensus.cc:3058] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.324939 21779 raft_consensus.cc:3058] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.325919 21779 raft_consensus.cc:2466] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:40.325919 21637 raft_consensus.cc:2466] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:40.326170 21853 leader_election.cc:304] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac33cb8d699d474493d7169da742f701, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:40.326324 22062 raft_consensus.cc:2802] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:40.326407 22062 raft_consensus.cc:695] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Becoming Leader. State: Replica: ac33cb8d699d474493d7169da742f701, State: Running, Role: LEADER
I20250903 14:26:40.326542 22062 consensus_queue.cc:237] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.327955 21477 catalog_manager.cc:5582] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac33cb8d699d474493d7169da742f701 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } health_report { overall_health: HEALTHY } } }
I20250903 14:26:40.345073 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:40.342740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.19.65.131:42449)
0903 14:26:40.342958 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:40.342961 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:40.342970 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:26:40.342998 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:40.343000 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:40.343033 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:40.343097 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:26:40.343440 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:40.343929 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:40.344521 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:40.344674 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:40.344710 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:26:40.344764 (+ 54us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:26:40.344821 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:40.344914 (+ 93us) server_negotiation.cc:300] Negotiation successful
0903 14:26:40.344954 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20250903 14:26:40.349668 21779 raft_consensus.cc:1273] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:40.349689 21637 raft_consensus.cc:1273] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:40.349987 22051 consensus_queue.cc:1035] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:40.350075 22064 consensus_queue.cc:1035] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:40.375984 21477 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:40.376089 21477 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:40.377763 21477 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Sending DeleteTablet for 3 replicas of tablet 41c67b85399448918af0a25fe17c7efe
I20250903 14:26:40.378196 21616 tablet_service.cc:1515] Processing DeleteTablet for tablet 41c67b85399448918af0a25fe17c7efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46034
I20250903 14:26:40.378264 21900 tablet_service.cc:1515] Processing DeleteTablet for tablet 41c67b85399448918af0a25fe17c7efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55886
I20250903 14:26:40.378288 21759 tablet_service.cc:1515] Processing DeleteTablet for tablet 41c67b85399448918af0a25fe17c7efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:40 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59562
I20250903 14:26:40.378372 22069 tablet_replica.cc:331] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: stopping tablet replica
I20250903 14:26:40.378440 22070 tablet_replica.cc:331] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: stopping tablet replica
I20250903 14:26:40.378472 22069 raft_consensus.cc:2241] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:40.378546 22070 raft_consensus.cc:2241] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:40.378562 22069 raft_consensus.cc:2270] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.378618 22070 raft_consensus.cc:2270] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.378693 22071 tablet_replica.cc:331] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: stopping tablet replica
I20250903 14:26:40.378798 22071 raft_consensus.cc:2241] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:40.378938 22071 raft_consensus.cc:2270] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:40.378930 21477 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:26:40.379267 21477 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:26:40.379324 22070 ts_tablet_manager.cc:1905] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.379628 22069 ts_tablet_manager.cc:1905] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.381458 22070 ts_tablet_manager.cc:1918] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:40.381589 22070 log.cc:1199] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/41c67b85399448918af0a25fe17c7efe
I20250903 14:26:40.381678 22069 ts_tablet_manager.cc:1918] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:40.381736 22069 log.cc:1199] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/41c67b85399448918af0a25fe17c7efe
I20250903 14:26:40.381824 22070 ts_tablet_manager.cc:1939] T 41c67b85399448918af0a25fe17c7efe P 533c1866188445b18694d58577c087ba: Deleting consensus metadata
I20250903 14:26:40.381974 22069 ts_tablet_manager.cc:1939] T 41c67b85399448918af0a25fe17c7efe P e16c81c2c821459488c8caddcbcb5ec7: Deleting consensus metadata
I20250903 14:26:40.382405 21463 catalog_manager.cc:4928] TS 533c1866188445b18694d58577c087ba (127.19.65.130:42763): tablet 41c67b85399448918af0a25fe17c7efe (table test-table [id=d7b2209c3383454b98c44b918e93f9b3]) successfully deleted
I20250903 14:26:40.382622 22071 ts_tablet_manager.cc:1905] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:40.383078 21463 catalog_manager.cc:4928] TS e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021): tablet 41c67b85399448918af0a25fe17c7efe (table test-table [id=d7b2209c3383454b98c44b918e93f9b3]) successfully deleted
I20250903 14:26:40.384660 21616 tablet_service.cc:1468] Processing CreateTablet for tablet 203e93e26fb64d458119d15fdab365f8 (DEFAULT_TABLE table=test-table [id=edb5d11efe2f42bf91789acb4b850704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.384747 21759 tablet_service.cc:1468] Processing CreateTablet for tablet 203e93e26fb64d458119d15fdab365f8 (DEFAULT_TABLE table=test-table [id=edb5d11efe2f42bf91789acb4b850704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.384877 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203e93e26fb64d458119d15fdab365f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.384893 22071 ts_tablet_manager.cc:1918] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:26:40.384933 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203e93e26fb64d458119d15fdab365f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.384953 22071 log.cc:1199] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/41c67b85399448918af0a25fe17c7efe
I20250903 14:26:40.385136 22071 ts_tablet_manager.cc:1939] T 41c67b85399448918af0a25fe17c7efe P ac33cb8d699d474493d7169da742f701: Deleting consensus metadata
I20250903 14:26:40.385598 21462 catalog_manager.cc:4928] TS ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449): tablet 41c67b85399448918af0a25fe17c7efe (table test-table [id=d7b2209c3383454b98c44b918e93f9b3]) successfully deleted
I20250903 14:26:40.385708 21900 tablet_service.cc:1468] Processing CreateTablet for tablet 203e93e26fb64d458119d15fdab365f8 (DEFAULT_TABLE table=test-table [id=edb5d11efe2f42bf91789acb4b850704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:26:40.385895 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203e93e26fb64d458119d15fdab365f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:40.386638 22081 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Bootstrap starting.
I20250903 14:26:40.387351 22081 tablet_bootstrap.cc:654] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.387420 22082 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Bootstrap starting.
I20250903 14:26:40.387796 22080 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: Bootstrap starting.
I20250903 14:26:40.388010 22082 tablet_bootstrap.cc:654] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.388381 22081 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: No bootstrap required, opened a new log
I20250903 14:26:40.388460 22081 ts_tablet_manager.cc:1397] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250903 14:26:40.388754 22080 tablet_bootstrap.cc:654] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:40.388824 22081 raft_consensus.cc:357] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.388908 22081 raft_consensus.cc:383] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.388939 22081 raft_consensus.cc:738] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c1866188445b18694d58577c087ba, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.389016 22082 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: No bootstrap required, opened a new log
I20250903 14:26:40.389077 22082 ts_tablet_manager.cc:1397] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:40.389048 22081 consensus_queue.cc:260] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.389290 22081 ts_tablet_manager.cc:1428] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:26:40.389492 22082 raft_consensus.cc:357] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.389590 22082 raft_consensus.cc:383] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.389619 22082 raft_consensus.cc:738] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33cb8d699d474493d7169da742f701, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.389727 22080 tablet_bootstrap.cc:492] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: No bootstrap required, opened a new log
I20250903 14:26:40.389708 22082 consensus_queue.cc:260] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.389793 22080 ts_tablet_manager.cc:1397] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:40.389915 22082 ts_tablet_manager.cc:1428] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:40.390146 22080 raft_consensus.cc:357] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.390229 22080 raft_consensus.cc:383] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:40.390276 22080 raft_consensus.cc:738] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16c81c2c821459488c8caddcbcb5ec7, State: Initialized, Role: FOLLOWER
I20250903 14:26:40.390381 22080 consensus_queue.cc:260] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.390620 22080 ts_tablet_manager.cc:1428] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:40.442229 22062 raft_consensus.cc:491] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:40.442324 22062 raft_consensus.cc:513] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.442673 22062 leader_election.cc:290] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:40.442965 21636 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "203e93e26fb64d458119d15fdab365f8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7" is_pre_election: true
I20250903 14:26:40.443096 21636 raft_consensus.cc:2466] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:40.443234 21778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "203e93e26fb64d458119d15fdab365f8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba" is_pre_election: true
I20250903 14:26:40.443365 21778 raft_consensus.cc:2466] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 0.
I20250903 14:26:40.443414 21853 leader_election.cc:304] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac33cb8d699d474493d7169da742f701, e16c81c2c821459488c8caddcbcb5ec7; no voters:
I20250903 14:26:40.443565 22062 raft_consensus.cc:2802] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:26:40.443624 22062 raft_consensus.cc:491] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:26:40.443662 22062 raft_consensus.cc:3058] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.444506 22062 raft_consensus.cc:513] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.444761 22062 leader_election.cc:290] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Requested vote from peers e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021), 533c1866188445b18694d58577c087ba (127.19.65.130:42763)
I20250903 14:26:40.445036 21636 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "203e93e26fb64d458119d15fdab365f8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16c81c2c821459488c8caddcbcb5ec7"
I20250903 14:26:40.445073 21778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "203e93e26fb64d458119d15fdab365f8" candidate_uuid: "ac33cb8d699d474493d7169da742f701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533c1866188445b18694d58577c087ba"
I20250903 14:26:40.445117 21636 raft_consensus.cc:3058] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.445171 21778 raft_consensus.cc:3058] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:40.445950 21636 raft_consensus.cc:2466] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:40.445981 21778 raft_consensus.cc:2466] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac33cb8d699d474493d7169da742f701 in term 1.
I20250903 14:26:40.446220 21853 leader_election.cc:304] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533c1866188445b18694d58577c087ba, ac33cb8d699d474493d7169da742f701; no voters:
I20250903 14:26:40.446368 22062 raft_consensus.cc:2802] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:40.446441 22062 raft_consensus.cc:695] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Becoming Leader. State: Replica: ac33cb8d699d474493d7169da742f701, State: Running, Role: LEADER
I20250903 14:26:40.446561 22062 consensus_queue.cc:237] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } }
I20250903 14:26:40.447839 21478 catalog_manager.cc:5582] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac33cb8d699d474493d7169da742f701 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "ac33cb8d699d474493d7169da742f701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac33cb8d699d474493d7169da742f701" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42449 } health_report { overall_health: HEALTHY } } }
I20250903 14:26:40.474890 21637 raft_consensus.cc:1273] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:40.475101 21778 raft_consensus.cc:1273] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Refusing update from remote peer ac33cb8d699d474493d7169da742f701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:26:40.475447 22062 consensus_queue.cc:1035] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533c1866188445b18694d58577c087ba" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:26:40.475560 22047 consensus_queue.cc:1035] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16c81c2c821459488c8caddcbcb5ec7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 14:26:40.486282 22049 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250903 14:26:42.025624 21478 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:42.025753 21478 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:26:42.027699 21478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9eae0607571245ceba488d7aeaa520c1: Sending DeleteTablet for 3 replicas of tablet 203e93e26fb64d458119d15fdab365f8
I20250903 14:26:42.028128 21616 tablet_service.cc:1515] Processing DeleteTablet for tablet 203e93e26fb64d458119d15fdab365f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:42 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46034
I20250903 14:26:42.028183 21900 tablet_service.cc:1515] Processing DeleteTablet for tablet 203e93e26fb64d458119d15fdab365f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:42 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55886
I20250903 14:26:42.028204 21759 tablet_service.cc:1515] Processing DeleteTablet for tablet 203e93e26fb64d458119d15fdab365f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:26:42 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59562
I20250903 14:26:42.028513 22119 tablet_replica.cc:331] T 203e93e26fb64d458119d15fdab365f8 P e16c81c2c821459488c8caddcbcb5ec7: stopping tablet replica
I20250903 14:26:42.028512 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21552
I20250903 14:26:42.029106 22120 tablet_replica.cc:331] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: stopping tablet replica
I20250903 14:26:42.028558 22121 tablet_replica.cc:331] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: stopping tablet replica
I20250903 14:26:42.029285 22120 raft_consensus.cc:2241] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:26:42.029350 22121 raft_consensus.cc:2241] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:26:42.029500 22121 raft_consensus.cc:2270] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:42.029508 22120 raft_consensus.cc:2270] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:26:42.029958 22121 ts_tablet_manager.cc:1905] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:26:42.031934 22121 ts_tablet_manager.cc:1918] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 14:26:42.031996 22121 log.cc:1199] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/203e93e26fb64d458119d15fdab365f8
I20250903 14:26:42.032258 22121 ts_tablet_manager.cc:1939] T 203e93e26fb64d458119d15fdab365f8 P 533c1866188445b18694d58577c087ba: Deleting consensus metadata
I20250903 14:26:42.032753 21463 catalog_manager.cc:4928] TS 533c1866188445b18694d58577c087ba (127.19.65.130:42763): tablet 203e93e26fb64d458119d15fdab365f8 (table test-table [id=edb5d11efe2f42bf91789acb4b850704]) successfully deleted
I20250903 14:26:42.033829 22120 ts_tablet_manager.cc:1905] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 14:26:42.034873 21463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:46021 (error 108)
I20250903 14:26:42.035148 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21694
I20250903 14:26:42.036037 22120 ts_tablet_manager.cc:1918] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 14:26:42.036053 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:42.035653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46021 (local address 127.0.0.1:57768)
0903 14:26:42.035890 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:42.035961 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20250903 14:26:42.036123 22120 log.cc:1199] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/203e93e26fb64d458119d15fdab365f8
I20250903 14:26:42.036348 22120 ts_tablet_manager.cc:1939] T 203e93e26fb64d458119d15fdab365f8 P ac33cb8d699d474493d7169da742f701: Deleting consensus metadata
W20250903 14:26:42.036307 21463 catalog_manager.cc:4671] TS e16c81c2c821459488c8caddcbcb5ec7 (127.19.65.129:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 203e93e26fb64d458119d15fdab365f8: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46021: connect: Connection refused (error 111)
I20250903 14:26:42.036887 21462 catalog_manager.cc:4928] TS ac33cb8d699d474493d7169da742f701 (127.19.65.131:42449): tablet 203e93e26fb64d458119d15fdab365f8 (table test-table [id=edb5d11efe2f42bf91789acb4b850704]) successfully deleted
I20250903 14:26:42.042053 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21836
I20250903 14:26:42.049115 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 21446
2025-09-03T14:26:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6834 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:42 dist-test-slave-4wpb krb5kdc[22126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:42 dist-test-slave-4wpb krb5kdc[22126](info): set up 2 sockets
Sep 03 14:26:42 dist-test-slave-4wpb krb5kdc[22126](info): commencing operation
krb5kdc: starting...
W20250903 14:26:43.833328 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.763s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:43 dist-test-slave-4wpb krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:43.941617 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45141
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:44.029125 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:44.029268 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:44.029310 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:44.031848 22142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:44.031896 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:44.031915 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:44.031931 22142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:44.031945 22142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:44.035517 22142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:44.036227 22142 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:44.036851 22142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:44.041378 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:44.041332 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.041447 22142 server_base.cc:1047] running on GCE node
W20250903 14:26:44.041337 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.041954 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:44.042610 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:44.043744 22142 hybrid_clock.cc:648] HybridClock initialized: now 1756909604043740 us; error 22 us; skew 500 ppm
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:44.046051 22142 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:44.046804 22142 webserver.cc:480] Webserver started at http://127.19.65.190:45253/ using document root <none> and password file <none>
I20250903 14:26:44.047214 22142 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:44.047260 22142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:44.047400 22142 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:44.048694 22142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "c59c7093876441d9a32ab2f7e088fe3f"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "652a01da54d63c2ae4e4ed5bed9dc35d"
server_key_iv: "0ffc5d7c4cc11c09fcc95d6e36a23626"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.049072 22142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "c59c7093876441d9a32ab2f7e088fe3f"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "652a01da54d63c2ae4e4ed5bed9dc35d"
server_key_iv: "0ffc5d7c4cc11c09fcc95d6e36a23626"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.051693 22142 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:26:44.053362 22157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.054217 22142 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:44.054342 22142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "c59c7093876441d9a32ab2f7e088fe3f"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "652a01da54d63c2ae4e4ed5bed9dc35d"
server_key_iv: "0ffc5d7c4cc11c09fcc95d6e36a23626"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.054407 22142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:44.068877 22142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:44.074810 22142 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:44.074942 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:44.080360 22209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:45141 every 8 connection(s)
I20250903 14:26:44.080364 22142 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45141
I20250903 14:26:44.081111 22142 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:44.083087 22210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:44.085440 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22142
I20250903 14:26:44.085566 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:44.085784 19718 external_mini_cluster.cc:1468] Setting key 4f002bf07efc1600cecec771c7b7e977
I20250903 14:26:44.088263 22210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f: Bootstrap starting.
I20250903 14:26:44.090232 22210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:44.090844 22210 log.cc:826] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f: Log is configured to *not* fsync() on all Append() calls
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:44.092491 22210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f: No bootstrap required, opened a new log
I20250903 14:26:44.095372 22210 raft_consensus.cc:357] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } }
I20250903 14:26:44.095548 22210 raft_consensus.cc:383] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:44.095609 22210 raft_consensus.cc:738] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c7093876441d9a32ab2f7e088fe3f, State: Initialized, Role: FOLLOWER
I20250903 14:26:44.095974 22210 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } }
I20250903 14:26:44.096098 22210 raft_consensus.cc:397] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:44.096200 22210 raft_consensus.cc:491] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:44.096302 22210 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:44.096915 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.086723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.19.65.190:45141)
0903 14:26:44.087041 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:44.087050 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:44.087083 (+ 33us) server_negotiation.cc:408] Connection header received
0903 14:26:44.087619 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:44.087640 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:44.087932 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:44.088199 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:44.089066 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.090063 (+ 997us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.090623 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.090849 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.092837 (+ 1988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:44.092852 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:44.092862 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:44.092901 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:44.094765 (+ 1864us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.095130 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.095135 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.095139 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.095214 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.095404 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.095407 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.095408 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.095727 (+ 319us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:44.095893 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:44.096103 (+ 210us) server_negotiation.cc:300] Negotiation successful
0903 14:26:44.096316 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20250903 14:26:44.097066 22210 raft_consensus.cc:513] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } }
I20250903 14:26:44.097357 22210 leader_election.cc:304] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59c7093876441d9a32ab2f7e088fe3f; no voters:
I20250903 14:26:44.097607 22210 leader_election.cc:290] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:44.097697 22215 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:44.098788 22215 raft_consensus.cc:695] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [term 1 LEADER]: Becoming Leader. State: Replica: c59c7093876441d9a32ab2f7e088fe3f, State: Running, Role: LEADER
I20250903 14:26:44.099071 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } }
I20250903 14:26:44.099354 22210 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:44.099886 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59c7093876441d9a32ab2f7e088fe3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } } }
I20250903 14:26:44.099975 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:44.100160 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59c7093876441d9a32ab2f7e088fe3f. Latest consensus state: current_term: 1 leader_uuid: "c59c7093876441d9a32ab2f7e088fe3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59c7093876441d9a32ab2f7e088fe3f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45141 } } }
I20250903 14:26:44.100234 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:44.100474 22224 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:44.102515 22224 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:44.106563 22224 catalog_manager.cc:1349] Generated new cluster ID: fefa8952ab3745a6a1af70fbb346ede5
I20250903 14:26:44.106617 22224 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:44.116514 22224 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:44.117146 22224 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:44.124011 22224 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c59c7093876441d9a32ab2f7e088fe3f: Generated new TSK 0
I20250903 14:26:44.124539 22224 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:44.170127 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45141
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:44.258145 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:44.258296 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:44.258364 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:44.260998 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:44.261042 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:44.261104 22238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:44.264688 22238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:44.265441 22238 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:44.266104 22238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:44.271024 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:44.271066 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.271205 22238 server_base.cc:1047] running on GCE node
W20250903 14:26:44.271098 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.271539 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:44.271957 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:44.273124 22238 hybrid_clock.cc:648] HybridClock initialized: now 1756909604273105 us; error 25 us; skew 500 ppm
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:44.275357 22238 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:44.276093 22238 webserver.cc:480] Webserver started at http://127.19.65.129:42255/ using document root <none> and password file <none>
I20250903 14:26:44.276490 22238 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:44.276527 22238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:44.276654 22238 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:44.277993 22238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "c548b0fb3932430f9aab2031c9b620ed"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "2be81ecc1ac4ebede219e267fdc4292c"
server_key_iv: "0ecd076754907fff7a1f4a292f8eb392"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.278357 22238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "c548b0fb3932430f9aab2031c9b620ed"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "2be81ecc1ac4ebede219e267fdc4292c"
server_key_iv: "0ecd076754907fff7a1f4a292f8eb392"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.280854 22238 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250903 14:26:44.282532 22253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.283324 22238 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 14:26:44.283407 22238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "c548b0fb3932430f9aab2031c9b620ed"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "2be81ecc1ac4ebede219e267fdc4292c"
server_key_iv: "0ecd076754907fff7a1f4a292f8eb392"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.283493 22238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:44.292603 22238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:44.294767 22238 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:44.294917 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:44.295331 22238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:44.295998 22238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:44.296039 22238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.296082 22238 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:44.296114 22238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.303258 22238 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:39961
I20250903 14:26:44.303287 22366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:39961 every 8 connection(s)
I20250903 14:26:44.304001 22238 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:44.313758 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22238
I20250903 14:26:44.313865 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:44.314085 19718 external_mini_cluster.cc:1468] Setting key 01c234e630eec1c7c833c84dd7ee0306
I20250903 14:26:44.314144 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.305372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:57007 (local address 127.19.65.190:45141)
0903 14:26:44.305499 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:44.305503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:44.305969 (+ 466us) server_negotiation.cc:408] Connection header received
0903 14:26:44.306597 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:44.306600 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:44.306656 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:44.306727 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:44.307676 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.308144 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.309202 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.309398 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.311374 (+ 1976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:44.311393 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:44.311398 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:44.311421 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:44.312453 (+ 1032us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.312840 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.312843 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.312844 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.312882 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.313131 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.313135 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.313137 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.313275 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:44.313368 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:44.313854 (+ 486us) server_negotiation.cc:300] Negotiation successful
0903 14:26:44.313988 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:26:44.314643 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.305537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45141 (local address 127.19.65.129:57007)
0903 14:26:44.305850 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:44.305886 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:44.306434 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:44.306856 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:44.306863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:44.307153 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:44.307553 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.307563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.308262 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.308265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:44.309060 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.309069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.309256 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.309806 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:44.309859 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:44.311233 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:44.312550 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.312554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.312563 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.312749 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.312955 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.312957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.312959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.313023 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.313380 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:44.313388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:44.313630 (+ 242us) client_negotiation.cc:769] Sending connection context
0903 14:26:44.313798 (+ 168us) client_negotiation.cc:240] Negotiation successful
0903 14:26:44.314018 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20250903 14:26:44.315627 22367 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45141
I20250903 14:26:44.315817 22367 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:44.316215 22367 heartbeater.cc:507] Master 127.19.65.190:45141 requested a full tablet report, sending...
I20250903 14:26:44.317579 22173 ts_manager.cc:194] Registered new tserver with Master: c548b0fb3932430f9aab2031c9b620ed (127.19.65.129:39961)
I20250903 14:26:44.318578 22173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:57007
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:44.349946 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45141
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:44.437525 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:44.437677 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:44.437731 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:44.440371 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:44.440415 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:44.440519 22374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:44.444120 22374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:44.444927 22374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:44.445569 22374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:44.450518 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.450598 22374 server_base.cc:1047] running on GCE node
W20250903 14:26:44.450539 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:44.450528 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.450980 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:44.451406 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:44.452538 22374 hybrid_clock.cc:648] HybridClock initialized: now 1756909604452519 us; error 30 us; skew 500 ppm
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:44.454952 22374 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:44.455785 22374 webserver.cc:480] Webserver started at http://127.19.65.130:46315/ using document root <none> and password file <none>
I20250903 14:26:44.456259 22374 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:44.456300 22374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:44.456449 22374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:44.457852 22374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "7d7f584d7c664e0abc3164ff829031f0"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "98568ee0788579c90511dfd2606db79e"
server_key_iv: "e51eb867e45054ea6d745a9fa51cd449"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.458232 22374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "7d7f584d7c664e0abc3164ff829031f0"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "98568ee0788579c90511dfd2606db79e"
server_key_iv: "e51eb867e45054ea6d745a9fa51cd449"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.460747 22374 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:26:44.462430 22389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.463217 22374 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:44.463310 22374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "7d7f584d7c664e0abc3164ff829031f0"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "98568ee0788579c90511dfd2606db79e"
server_key_iv: "e51eb867e45054ea6d745a9fa51cd449"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.463388 22374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:44.476202 22374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:44.478426 22374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:44.478564 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:44.478953 22374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:44.479578 22374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:44.479620 22374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.479666 22374 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:44.479696 22374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.486963 22374 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45023
I20250903 14:26:44.486982 22502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:45023 every 8 connection(s)
I20250903 14:26:44.487686 22374 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:26:44.493453 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22374
I20250903 14:26:44.493597 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:44.493875 19718 external_mini_cluster.cc:1468] Setting key b27ca4ca52af53e32f3bf5f84a479db4
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:44.497962 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.488962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57775 (local address 127.19.65.190:45141)
0903 14:26:44.489081 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:44.489085 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:44.489665 (+ 580us) server_negotiation.cc:408] Connection header received
0903 14:26:44.490392 (+ 727us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:44.490395 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:44.490456 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:44.490522 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:44.491495 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.491923 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.492853 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.493015 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.495231 (+ 2216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:44.495247 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:44.495250 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:44.495276 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:44.496460 (+ 1184us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.496856 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.496859 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.496860 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.496898 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.497114 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.497117 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.497119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.497238 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:44.497312 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:44.497724 (+ 412us) server_negotiation.cc:300] Negotiation successful
0903 14:26:44.497823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 14:26:44.498409 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.489157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45141 (local address 127.19.65.130:57775)
0903 14:26:44.489557 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:44.489587 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:44.490222 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:44.490655 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:44.490661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:44.490942 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:44.491367 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.491377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.492022 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.492024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:44.492733 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.492740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.492905 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.493434 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:44.493468 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:44.495075 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:44.496567 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.496571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.496580 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.496766 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.496968 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.496973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.496975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.497038 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.497323 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:44.497330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:44.497522 (+ 192us) client_negotiation.cc:769] Sending connection context
0903 14:26:44.497681 (+ 159us) client_negotiation.cc:240] Negotiation successful
0903 14:26:44.497877 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20250903 14:26:44.499289 22503 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45141
I20250903 14:26:44.499462 22503 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:44.499823 22503 heartbeater.cc:507] Master 127.19.65.190:45141 requested a full tablet report, sending...
I20250903 14:26:44.500636 22173 ts_manager.cc:194] Registered new tserver with Master: 7d7f584d7c664e0abc3164ff829031f0 (127.19.65.130:45023)
I20250903 14:26:44.501081 22173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:57775
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:44.530268 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45141
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:44.616492 22510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:44.616647 22510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:44.616719 22510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:44.619320 22510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:44.619369 22510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:44.619457 22510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:44.622725 22510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:44.623510 22510 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:44.624195 22510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:44.629179 22515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.629248 22510 server_base.cc:1047] running on GCE node
W20250903 14:26:44.629280 22518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:44.629179 22516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:44.629699 22510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:44.630133 22510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:44.631269 22510 hybrid_clock.cc:648] HybridClock initialized: now 1756909604631260 us; error 28 us; skew 500 ppm
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:44.633602 22510 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:44.634439 22510 webserver.cc:480] Webserver started at http://127.19.65.131:33399/ using document root <none> and password file <none>
I20250903 14:26:44.634892 22510 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:44.634938 22510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:44.635097 22510 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:44.636444 22510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "18e9be2ea3e54891b164f3755c7995e1"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "e3f815f240bcb17ada09f6e3e345d677"
server_key_iv: "5be124ed2e23738fd4a822143f01c2a0"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.636818 22510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "18e9be2ea3e54891b164f3755c7995e1"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "e3f815f240bcb17ada09f6e3e345d677"
server_key_iv: "5be124ed2e23738fd4a822143f01c2a0"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.639496 22510 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:26:44.641124 22525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.642014 22510 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:44.642104 22510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "18e9be2ea3e54891b164f3755c7995e1"
format_stamp: "Formatted at 2025-09-03 14:26:44 on dist-test-slave-4wpb"
server_key: "e3f815f240bcb17ada09f6e3e345d677"
server_key_iv: "5be124ed2e23738fd4a822143f01c2a0"
server_key_version: "encryptionkey@0"
I20250903 14:26:44.642179 22510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:44.660987 22510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:44.663118 22510 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:44.663267 22510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:44.663702 22510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:44.664413 22510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:44.664453 22510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.664489 22510 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:44.664516 22510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:44.671639 22510 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:46057
I20250903 14:26:44.671681 22638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:46057 every 8 connection(s)
I20250903 14:26:44.672317 22510 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:44.673887 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22510
I20250903 14:26:44.673991 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:44.674271 19718 external_mini_cluster.cc:1468] Setting key c9d23fd86a969b50f023dcc9c96ffc5d
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909604, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:44.682010 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.673706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:49591 (local address 127.19.65.190:45141)
0903 14:26:44.673816 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:44.673819 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:44.674502 (+ 683us) server_negotiation.cc:408] Connection header received
0903 14:26:44.675298 (+ 796us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:44.675301 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:44.675355 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:44.675419 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:44.676397 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.676847 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.677810 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.677977 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.679524 (+ 1547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:44.679537 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:44.679539 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:44.679559 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:44.680536 (+ 977us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.680905 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.680908 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.680910 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.680946 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:44.681164 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:44.681167 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:44.681169 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:44.681291 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:44.681368 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:44.681763 (+ 395us) server_negotiation.cc:300] Negotiation successful
0903 14:26:44.681881 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:26:44.682428 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:44.673969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45141 (local address 127.19.65.131:49591)
0903 14:26:44.674399 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:44.674427 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:44.675139 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:44.675537 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:44.675544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:44.675838 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:44.676277 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.676287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.676945 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:44.676948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:44.677707 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:44.677715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:44.677868 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:44.678290 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:44.678308 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:44.679406 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:44.680628 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.680632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.680641 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.680822 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.681015 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:44.681020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:44.681022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:44.681091 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:44.681375 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:44.681382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:44.681575 (+ 193us) client_negotiation.cc:769] Sending connection context
0903 14:26:44.681720 (+ 145us) client_negotiation.cc:240] Negotiation successful
0903 14:26:44.681913 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20250903 14:26:44.683215 22639 heartbeater.cc:344] Connected to a master server at 127.19.65.190:45141
I20250903 14:26:44.683404 22639 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:44.683799 22639 heartbeater.cc:507] Master 127.19.65.190:45141 requested a full tablet report, sending...
I20250903 14:26:44.684604 22173 ts_manager.cc:194] Registered new tserver with Master: 18e9be2ea3e54891b164f3755c7995e1 (127.19.65.131:46057)
I20250903 14:26:44.685053 22173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:49591
I20250903 14:26:44.687206 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 14:26:44.696943 22213 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:26:44.696037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.19.65.190:45141)
0903 14:26:44.696133 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:44.696136 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:44.696262 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:26:44.696336 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:44.696338 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:44.696366 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:44.696423 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:44.696871 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40738: BlockingRecv error: recv got EOF from 127.0.0.1:40738 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:26:44.697794 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22238
I20250903 14:26:44.702814 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22374
I20250903 14:26:44.707579 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22510
I20250903 14:26:44.712142 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22142
2025-09-03T14:26:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2662 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22651](info): set up 2 sockets
Sep 03 14:26:44 dist-test-slave-4wpb krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20250903 14:26:46.461318 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.729s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:46 dist-test-slave-4wpb krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:46.569334 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34865
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:46.657721 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:46.657891 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:46.657965 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:46.660504 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:46.660553 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:46.660588 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:46.660624 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:46.660661 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:46.664254 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:46.665014 22667 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:46.665655 22667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:46.669854 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:46.669888 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:46.669865 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:46.669991 22667 server_base.cc:1047] running on GCE node
I20250903 14:26:46.670401 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:46.671075 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:46.672224 22667 hybrid_clock.cc:648] HybridClock initialized: now 1756909606672212 us; error 24 us; skew 500 ppm
Sep 03 14:26:46 dist-test-slave-4wpb krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909606, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:46.674474 22667 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:46.675274 22667 webserver.cc:480] Webserver started at http://127.19.65.190:33883/ using document root <none> and password file <none>
I20250903 14:26:46.675714 22667 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:46.675761 22667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:46.675912 22667 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:46.677284 22667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "9b3703e2db4749928eb33da0ad6cfcfb"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "e604a210faa4323602da4dabcaf8fc45"
server_key_iv: "32afb09d1911bec810a104412e7c48b5"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.677680 22667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "9b3703e2db4749928eb33da0ad6cfcfb"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "e604a210faa4323602da4dabcaf8fc45"
server_key_iv: "32afb09d1911bec810a104412e7c48b5"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.680286 22667 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:26:46.681918 22682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:46.682735 22667 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:46.682821 22667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "9b3703e2db4749928eb33da0ad6cfcfb"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "e604a210faa4323602da4dabcaf8fc45"
server_key_iv: "32afb09d1911bec810a104412e7c48b5"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.682884 22667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:46.691898 22667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:46.694142 22667 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:46.694305 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:46.699828 22734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:34865 every 8 connection(s)
I20250903 14:26:46.699828 22667 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:34865
I20250903 14:26:46.700568 22667 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:46.702634 22735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:46.702880 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22667
I20250903 14:26:46.702978 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:46.703202 19718 external_mini_cluster.cc:1468] Setting key cc2e883ad08e181c28f06781e0d2d66f
I20250903 14:26:46.707213 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb: Bootstrap starting.
I20250903 14:26:46.709100 22735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:46.709671 22735 log.cc:826] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb: Log is configured to *not* fsync() on all Append() calls
Sep 03 14:26:46 dist-test-slave-4wpb krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:46.711145 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb: No bootstrap required, opened a new log
I20250903 14:26:46.713795 22735 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } }
I20250903 14:26:46.713979 22735 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:46.714042 22735 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3703e2db4749928eb33da0ad6cfcfb, State: Initialized, Role: FOLLOWER
I20250903 14:26:46.714336 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:46.704229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.19.65.190:34865)
0903 14:26:46.704896 (+ 667us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:46.704907 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:46.704940 (+ 33us) server_negotiation.cc:408] Connection header received
0903 14:26:46.705461 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:46.705498 (+ 37us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:46.705786 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:46.706063 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:46.707077 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.708018 (+ 941us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:46.708550 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.708774 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:46.710486 (+ 1712us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:46.710502 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:46.710512 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:46.710541 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:46.712193 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:46.712511 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:46.712516 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:46.712520 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:46.712588 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:46.712739 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:46.712741 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:46.712743 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:46.713086 (+ 343us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:46.713254 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:46.713431 (+ 177us) server_negotiation.cc:300] Negotiation successful
0903 14:26:46.713686 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":414,"threads_started":1}
I20250903 14:26:46.714447 22735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } }
I20250903 14:26:46.714565 22735 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:46.714617 22735 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:46.714748 22735 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:46.715525 22735 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } }
I20250903 14:26:46.715791 22735 leader_election.cc:304] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b3703e2db4749928eb33da0ad6cfcfb; no voters:
I20250903 14:26:46.716055 22735 leader_election.cc:290] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:46.716326 22740 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:46.716483 22740 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [term 1 LEADER]: Becoming Leader. State: Replica: 9b3703e2db4749928eb33da0ad6cfcfb, State: Running, Role: LEADER
I20250903 14:26:46.716763 22740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } }
I20250903 14:26:46.716840 22735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:46.718019 22741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } } }
I20250903 14:26:46.718046 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3703e2db4749928eb33da0ad6cfcfb. Latest consensus state: current_term: 1 leader_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3703e2db4749928eb33da0ad6cfcfb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34865 } } }
I20250903 14:26:46.718116 22741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:46.718122 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:46.718410 22751 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:46.720852 22751 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:46.724716 22751 catalog_manager.cc:1349] Generated new cluster ID: 0e1c565064364c26914ca97689d7967b
I20250903 14:26:46.724767 22751 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:46.732676 22751 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:46.733358 22751 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:46.742334 22751 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9b3703e2db4749928eb33da0ad6cfcfb: Generated new TSK 0
I20250903 14:26:46.742983 22751 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:46.780038 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34865
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:46.868355 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:46.868502 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:46.868549 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:46.871218 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:46.871263 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:46.871353 22763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:46.874970 22763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:46.875766 22763 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:46.876426 22763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:46.881525 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:46.881507 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:46.881512 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:46.881783 22763 server_base.cc:1047] running on GCE node
I20250903 14:26:46.882063 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:46.882474 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:46.883605 22763 hybrid_clock.cc:648] HybridClock initialized: now 1756909606883595 us; error 22 us; skew 500 ppm
Sep 03 14:26:46 dist-test-slave-4wpb krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909606, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:46.885711 22763 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:46.886479 22763 webserver.cc:480] Webserver started at http://127.19.65.129:40577/ using document root <none> and password file <none>
I20250903 14:26:46.886946 22763 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:46.886991 22763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:46.887135 22763 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:46.888455 22763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "ed0e8e247d114c9e8ce5bd6162febe46"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "23b1060f232dcabf5c79aa2fd96e3ba1"
server_key_iv: "4d635a3150d3e1becd28e6e30b7f016b"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.888835 22763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "ed0e8e247d114c9e8ce5bd6162febe46"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "23b1060f232dcabf5c79aa2fd96e3ba1"
server_key_iv: "4d635a3150d3e1becd28e6e30b7f016b"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.891530 22763 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250903 14:26:46.893265 22778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:46.894165 22763 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:26:46.894261 22763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "ed0e8e247d114c9e8ce5bd6162febe46"
format_stamp: "Formatted at 2025-09-03 14:26:46 on dist-test-slave-4wpb"
server_key: "23b1060f232dcabf5c79aa2fd96e3ba1"
server_key_iv: "4d635a3150d3e1becd28e6e30b7f016b"
server_key_version: "encryptionkey@0"
I20250903 14:26:46.894340 22763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:46.902155 22763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:46.904304 22763 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:46.904438 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:46.904826 22763 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:46.905444 22763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:46.905507 22763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:46.905561 22763 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:46.905584 22763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:46.913144 22763 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:43829
I20250903 14:26:46.913175 22891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:43829 every 8 connection(s)
I20250903 14:26:46.913952 22763 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:46 dist-test-slave-4wpb krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909606, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:46.923642 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22763
I20250903 14:26:46.923754 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:46.923784 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:46.915414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:55123 (local address 127.19.65.190:34865)
0903 14:26:46.915526 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:46.915530 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:46.916068 (+ 538us) server_negotiation.cc:408] Connection header received
0903 14:26:46.916714 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:46.916717 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:46.916774 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:46.916839 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:46.918334 (+ 1495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.918785 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:46.919318 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.919482 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:46.921119 (+ 1637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:46.921134 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:46.921136 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:46.921154 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:46.922223 (+ 1069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:46.922597 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:46.922600 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:46.922601 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:46.922637 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:46.922863 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:46.922865 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:46.922867 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:46.922984 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:46.923092 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:46.923497 (+ 405us) server_negotiation.cc:300] Negotiation successful
0903 14:26:46.923617 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:26:46.924012 19718 external_mini_cluster.cc:1468] Setting key 099b2c250907e09576538005f344118b
I20250903 14:26:46.924266 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:46.915595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34865 (local address 127.19.65.129:55123)
0903 14:26:46.915956 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:46.915983 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:46.916547 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:46.916984 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:46.916990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:46.917271 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:46.918202 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:46.918215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.918894 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:46.918896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:46.919217 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:46.919223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:46.919374 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:46.919832 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:46.919850 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:46.920994 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:46.922317 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:46.922322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:46.922331 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:46.922513 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:46.922719 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:46.922721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:46.922723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:46.922788 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:46.923108 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:46.923115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:46.923288 (+ 173us) client_negotiation.cc:769] Sending connection context
0903 14:26:46.923450 (+ 162us) client_negotiation.cc:240] Negotiation successful
0903 14:26:46.923668 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20250903 14:26:46.925239 22892 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34865
I20250903 14:26:46.925426 22892 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:46.925927 22892 heartbeater.cc:507] Master 127.19.65.190:34865 requested a full tablet report, sending...
I20250903 14:26:46.927213 22699 ts_manager.cc:194] Registered new tserver with Master: ed0e8e247d114c9e8ce5bd6162febe46 (127.19.65.129:43829)
I20250903 14:26:46.928242 22699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:55123
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:46.961215 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34865
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:47.047353 22899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:47.047495 22899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:47.047540 22899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:47.050082 22899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:47.050123 22899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:47.050185 22899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:47.053406 22899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:47.054165 22899 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:47.054785 22899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:47.059702 22907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:47.059727 22905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:47.059893 22899 server_base.cc:1047] running on GCE node
W20250903 14:26:47.059736 22904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:47.060256 22899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:47.060633 22899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:47.061769 22899 hybrid_clock.cc:648] HybridClock initialized: now 1756909607061755 us; error 25 us; skew 500 ppm
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909607, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:47.063933 22899 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:47.064782 22899 webserver.cc:480] Webserver started at http://127.19.65.130:36443/ using document root <none> and password file <none>
I20250903 14:26:47.065191 22899 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:47.065235 22899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:47.065375 22899 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:47.066701 22899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "79598d3a62e246d2b7c7a1825948acad"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "034c39ec260766771bc27fc62f925fd7"
server_key_iv: "894f5d1e1b47d2a94913ee9bb58d070b"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.067082 22899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "79598d3a62e246d2b7c7a1825948acad"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "034c39ec260766771bc27fc62f925fd7"
server_key_iv: "894f5d1e1b47d2a94913ee9bb58d070b"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.069636 22899 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:26:47.071270 22914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.071992 22899 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:26:47.072075 22899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "79598d3a62e246d2b7c7a1825948acad"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "034c39ec260766771bc27fc62f925fd7"
server_key_iv: "894f5d1e1b47d2a94913ee9bb58d070b"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.072144 22899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:47.091642 22899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:47.093643 22899 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:47.093791 22899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:47.094271 22899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:47.094957 22899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:47.094996 22899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.095041 22899 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:47.095072 22899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.102466 22899 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34931
I20250903 14:26:47.102499 23027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:34931 every 8 connection(s)
I20250903 14:26:47.103174 22899 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:26:47.104817 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 22899
I20250903 14:26:47.104926 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:47.105115 19718 external_mini_cluster.cc:1468] Setting key 296613c60c2d4c5d31e855ec05b875fd
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909607, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20250903 14:26:47.113443 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:47.104563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:40931 (local address 127.19.65.190:34865)
0903 14:26:47.104709 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.104715 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.105179 (+ 464us) server_negotiation.cc:408] Connection header received
0903 14:26:47.105983 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.105987 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.106028 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.106105 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.107415 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.107861 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:47.108418 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.108595 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:47.110557 (+ 1962us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:47.110572 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:47.110574 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:47.110595 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:47.111866 (+ 1271us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:47.112362 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:47.112364 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:47.112365 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:47.112393 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:47.112640 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:47.112644 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:47.112645 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:47.112764 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:47.112829 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:47.113233 (+ 404us) server_negotiation.cc:300] Negotiation successful
0903 14:26:47.113318 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250903 14:26:47.113974 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:47.104723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34865 (local address 127.19.65.130:40931)
0903 14:26:47.105072 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:47.105105 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:47.105814 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:47.106252 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:47.106259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:47.106532 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:47.107285 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:47.107296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.107973 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:47.107976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:47.108317 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:47.108323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.108479 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:47.109049 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:47.109068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:47.110417 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:47.111979 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:47.111986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:47.111997 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:47.112279 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:47.112455 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:47.112458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:47.112460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:47.112555 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:47.112859 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:47.112866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:47.113035 (+ 169us) client_negotiation.cc:769] Sending connection context
0903 14:26:47.113204 (+ 169us) client_negotiation.cc:240] Negotiation successful
0903 14:26:47.113376 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20250903 14:26:47.114889 23028 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
I20250903 14:26:47.115092 23028 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:47.115504 23028 heartbeater.cc:507] Master 127.19.65.190:34865 requested a full tablet report, sending...
I20250903 14:26:47.116276 22699 ts_manager.cc:194] Registered new tserver with Master: 79598d3a62e246d2b7c7a1825948acad (127.19.65.130:34931)
I20250903 14:26:47.116735 22699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:40931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:47.142217 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34865
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:47.229104 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:47.229246 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:47.229295 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:47.231954 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:47.231998 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:47.232060 23035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:47.235734 23035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:47.236476 23035 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:47.237107 23035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:47.242367 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:47.242332 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:47.242334 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:47.246093 23035 server_base.cc:1047] running on GCE node
I20250903 14:26:47.246367 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:47.246773 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:47.247913 23035 hybrid_clock.cc:648] HybridClock initialized: now 1756909607247899 us; error 30 us; skew 500 ppm
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909607, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:47.249953 23035 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:47.250733 23035 webserver.cc:480] Webserver started at http://127.19.65.131:34779/ using document root <none> and password file <none>
I20250903 14:26:47.251138 23035 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:47.251180 23035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:47.251323 23035 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:47.252619 23035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "cc42148c18e64188b71737ead02e8fb4"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "75bb20447d23d979c474fa99ae8eff24"
server_key_iv: "53d9690cce5ef99649d748652d871168"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.253016 23035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "cc42148c18e64188b71737ead02e8fb4"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "75bb20447d23d979c474fa99ae8eff24"
server_key_iv: "53d9690cce5ef99649d748652d871168"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.255841 23035 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:26:47.257508 23050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.258271 23035 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:47.258344 23035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "cc42148c18e64188b71737ead02e8fb4"
format_stamp: "Formatted at 2025-09-03 14:26:47 on dist-test-slave-4wpb"
server_key: "75bb20447d23d979c474fa99ae8eff24"
server_key_iv: "53d9690cce5ef99649d748652d871168"
server_key_version: "encryptionkey@0"
I20250903 14:26:47.258421 23035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:47.276459 23035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:47.278429 23035 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:47.278558 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:47.278995 23035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:47.279609 23035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:47.279654 23035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.279690 23035 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:47.279716 23035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:47.287151 23035 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:42603
I20250903 14:26:47.287170 23163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:42603 every 8 connection(s)
I20250903 14:26:47.287835 23035 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909607, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:47.296036 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23035
I20250903 14:26:47.296125 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:47.296371 19718 external_mini_cluster.cc:1468] Setting key 5f910a6e5709f353ee5ed0b384a4d50e
I20250903 14:26:47.297587 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:47.289087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57591 (local address 127.19.65.190:34865)
0903 14:26:47.289216 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.289220 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.289739 (+ 519us) server_negotiation.cc:408] Connection header received
0903 14:26:47.290368 (+ 629us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.290370 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.290417 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.290481 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.291758 (+ 1277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.292202 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:47.292700 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.292811 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:47.294708 (+ 1897us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:47.294724 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:47.294726 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:47.294746 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:47.295802 (+ 1056us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:47.296288 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:47.296292 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:47.296294 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:47.296344 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:47.296586 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:47.296589 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:47.296591 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:47.296725 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:47.296829 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:47.297290 (+ 461us) server_negotiation.cc:300] Negotiation successful
0903 14:26:47.297429 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:26:47.298142 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:47.289277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34865 (local address 127.19.65.131:57591)
0903 14:26:47.289631 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:47.289659 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:47.290223 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:47.290594 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:47.290600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:47.290921 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:47.291656 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:47.291667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.292278 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:47.292281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:47.292625 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:47.292630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:47.292750 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:47.293232 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:47.293253 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:47.294516 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:47.295921 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:47.295925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:47.295931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:47.296201 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:47.296424 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:47.296429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:47.296430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:47.296507 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:47.296836 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:47.296840 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:47.297036 (+ 196us) client_negotiation.cc:769] Sending connection context
0903 14:26:47.297242 (+ 206us) client_negotiation.cc:240] Negotiation successful
0903 14:26:47.297429 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20250903 14:26:47.298976 23164 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34865
I20250903 14:26:47.299134 23164 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:47.299464 23164 heartbeater.cc:507] Master 127.19.65.190:34865 requested a full tablet report, sending...
I20250903 14:26:47.300232 22699 ts_manager.cc:194] Registered new tserver with Master: cc42148c18e64188b71737ead02e8fb4 (127.19.65.131:42603)
I20250903 14:26:47.300711 22699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:57591
I20250903 14:26:47.309698 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 14:26:47.416399 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:26:47.413667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.19.65.190:34865)
0903 14:26:47.413785 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.413789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.414541 (+ 752us) server_negotiation.cc:408] Connection header received
0903 14:26:47.415249 (+ 708us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.415252 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.415296 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.415362 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.416282 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41248: BlockingRecv error: recv got EOF from 127.0.0.1:41248 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20250903 14:26:47.417622 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:26:47.417033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.19.65.190:34865)
0903 14:26:47.417120 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.417123 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.417133 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:26:47.417162 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.417164 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.417196 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.417254 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.417531 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41260: BlockingRecv error: recv got EOF from 127.0.0.1:41260 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20250903 14:26:47.418426 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:26:47.417883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.19.65.190:34865)
0903 14:26:47.417968 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.417971 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.417981 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:26:47.418040 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.418043 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.418075 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.418130 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.418340 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41274: BlockingRecv error: recv got EOF from 127.0.0.1:41274 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20250903 14:26:47.421257 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:26:47.420774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.19.65.190:34865)
0903 14:26:47.420859 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:47.420861 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:47.420872 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:26:47.420896 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:47.420898 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:47.420925 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:47.420979 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:47.421206 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41288: BlockingRecv error: recv got EOF from 127.0.0.1:41288 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:26:47.478257 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22763
I20250903 14:26:47.482868 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22899
I20250903 14:26:47.487571 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23035
I20250903 14:26:47.491884 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 22667
2025-09-03T14:26:47Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2779 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[23187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[23187](info): set up 2 sockets
Sep 03 14:26:47 dist-test-slave-4wpb krb5kdc[23187](info): commencing operation
krb5kdc: starting...
W20250903 14:26:49.245604 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.733s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:49.357717 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44173
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:49.446432 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:49.446585 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:49.446640 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:49.449239 23203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:49.449283 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:49.449303 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:49.449342 23203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:49.449369 23203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:49.452785 23203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:49.453600 23203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:49.454309 23203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:49.458544 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:49.458510 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:49.458639 23203 server_base.cc:1047] running on GCE node
W20250903 14:26:49.458582 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:49.459018 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:49.459784 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:49.460927 23203 hybrid_clock.cc:648] HybridClock initialized: now 1756909609460907 us; error 33 us; skew 500 ppm
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909609, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:49.463214 23203 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:49.464110 23203 webserver.cc:480] Webserver started at http://127.19.65.190:35695/ using document root <none> and password file <none>
I20250903 14:26:49.464553 23203 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:49.464594 23203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:49.464742 23203 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:49.466087 23203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "6122c64daf8e4079a08bd218e84825e3"
format_stamp: "Formatted at 2025-09-03 14:26:49 on dist-test-slave-4wpb"
server_key: "20b665102cf5fafdca97520787d133de"
server_key_iv: "a747936a92379d8e709b4605ca0f9cf4"
server_key_version: "encryptionkey@0"
I20250903 14:26:49.466464 23203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "6122c64daf8e4079a08bd218e84825e3"
format_stamp: "Formatted at 2025-09-03 14:26:49 on dist-test-slave-4wpb"
server_key: "20b665102cf5fafdca97520787d133de"
server_key_iv: "a747936a92379d8e709b4605ca0f9cf4"
server_key_version: "encryptionkey@0"
I20250903 14:26:49.469028 23203 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:49.470714 23218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:49.471462 23203 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 14:26:49.471541 23203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "6122c64daf8e4079a08bd218e84825e3"
format_stamp: "Formatted at 2025-09-03 14:26:49 on dist-test-slave-4wpb"
server_key: "20b665102cf5fafdca97520787d133de"
server_key_iv: "a747936a92379d8e709b4605ca0f9cf4"
server_key_version: "encryptionkey@0"
I20250903 14:26:49.471616 23203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:49.481460 23203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:49.488483 23203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:49.488605 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:49.494516 23203 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44173
I20250903 14:26:49.494526 23270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:44173 every 8 connection(s)
I20250903 14:26:49.495239 23203 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:49.497131 23271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:49.501339 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: Bootstrap starting.
I20250903 14:26:49.501343 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23203
I20250903 14:26:49.501426 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:49.501636 19718 external_mini_cluster.cc:1468] Setting key 0a9c4f3a06dfd0d7e0bd782dadfb19f4
I20250903 14:26:49.503350 23271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:49.503998 23271 log.cc:826] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:49.505537 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: No bootstrap required, opened a new log
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:49.508060 23271 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } }
I20250903 14:26:49.508239 23271 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:49.508292 23271 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6122c64daf8e4079a08bd218e84825e3, State: Initialized, Role: FOLLOWER
I20250903 14:26:49.508644 23271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } }
I20250903 14:26:49.508744 23271 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:49.508775 23271 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:49.508836 23271 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:49.509454 23271 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } }
I20250903 14:26:49.509769 23271 leader_election.cc:304] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6122c64daf8e4079a08bd218e84825e3; no voters:
I20250903 14:26:49.509984 23271 leader_election.cc:290] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:49.510056 23276 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:49.510221 23276 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [term 1 LEADER]: Becoming Leader. State: Replica: 6122c64daf8e4079a08bd218e84825e3, State: Running, Role: LEADER
I20250903 14:26:49.510492 23276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } }
I20250903 14:26:49.510918 23271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:49.511967 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6122c64daf8e4079a08bd218e84825e3. Latest consensus state: current_term: 1 leader_uuid: "6122c64daf8e4079a08bd218e84825e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } } }
I20250903 14:26:49.511973 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6122c64daf8e4079a08bd218e84825e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6122c64daf8e4079a08bd218e84825e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44173 } } }
I20250903 14:26:49.512168 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:49.512194 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:49.513106 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:49.502637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.19.65.190:44173)
0903 14:26:49.502966 (+ 329us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:49.502973 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:49.503001 (+ 28us) server_negotiation.cc:408] Connection header received
0903 14:26:49.503464 (+ 463us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:49.503485 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:49.503750 (+ 265us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:49.503981 (+ 231us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:49.504881 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:49.505522 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:49.506238 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:49.506423 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:49.508496 (+ 2073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:49.508516 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:49.508527 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:49.508559 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:49.510333 (+ 1774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:49.510773 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:49.510778 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:49.510783 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:49.510846 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:49.511065 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:49.511067 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:49.511069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:49.511337 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:49.511496 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:49.512265 (+ 769us) server_negotiation.cc:300] Negotiation successful
0903 14:26:49.512462 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20250903 14:26:49.516139 19718 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20250903 14:26:49.516345 23291 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:26:49.516404 23291 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:26:49.516487 23285 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:49.517343 23285 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:49.522114 23285 catalog_manager.cc:1349] Generated new cluster ID: c1e5ed10047c46aba7a7541974cb0bf0
I20250903 14:26:49.522166 23285 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:49.530426 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:49.526824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.19.65.190:44173)
0903 14:26:49.526947 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:49.526951 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:49.527093 (+ 142us) server_negotiation.cc:408] Connection header received
0903 14:26:49.527173 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:49.527176 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:49.527214 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:49.527289 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:49.527826 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:49.528416 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:49.529102 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:49.529270 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:49.529648 (+ 378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:49.529661 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:49.529664 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:26:49.530054 (+ 390us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:49.530081 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:26:49.530096 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:49.530174 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:49.530281 (+ 107us) server_negotiation.cc:300] Negotiation successful
0903 14:26:49.530325 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:26:49.531333 23285 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:49.532227 23285 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:49.542516 23285 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6122c64daf8e4079a08bd218e84825e3: Generated new TSK 0
I20250903 14:26:49.543114 23285 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:26:49.549974 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23203
2025-09-03T14:26:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2055 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23304](info): set up 2 sockets
Sep 03 14:26:49 dist-test-slave-4wpb krb5kdc[23304](info): commencing operation
krb5kdc: starting...
W20250903 14:26:51.416052 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.848s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20250903 14:26:51.721889 19718 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.306s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20250903 14:26:51.882246 19718 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.160s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20250903 14:26:52.109640 19718 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.227s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/test-user.keytab.
W20250903 14:26:52.834995 19718 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.725s user 0.001s sys 0.001s
Sep 03 14:26:52 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
W20250903 14:26:53.766909 19718 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.19.65.190: real 0.915s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:53.785705 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42825
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:53.877067 23320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:53.877214 23320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:53.877257 23320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:53.879801 23320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:53.879843 23320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:53.879863 23320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:53.879877 23320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:53.879897 23320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:53.883173 23320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:53.883998 23320 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:53.884673 23320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:53.889053 23326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:53.889087 23325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:53.889055 23328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:53.889168 23320 server_base.cc:1047] running on GCE node
I20250903 14:26:53.889720 23320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:53.890424 23320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:53.891558 23320 hybrid_clock.cc:648] HybridClock initialized: now 1756909613891550 us; error 20 us; skew 500 ppm
Sep 03 14:26:53 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:53.893615 23320 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:53.894500 23320 webserver.cc:480] Webserver started at http://127.19.65.190:35601/ using document root <none> and password file <none>
I20250903 14:26:53.895002 23320 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:53.895048 23320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:53.895210 23320 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:53.896581 23320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "f6cfe423f36f406fa6593b9aa88f7018"
format_stamp: "Formatted at 2025-09-03 14:26:53 on dist-test-slave-4wpb"
server_key: "a538cce9fcd7ddfba9d18616176c20c9"
server_key_iv: "42c15737db66ac419852a3bfe13a6f49"
server_key_version: "encryptionkey@0"
I20250903 14:26:53.896968 23320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "f6cfe423f36f406fa6593b9aa88f7018"
format_stamp: "Formatted at 2025-09-03 14:26:53 on dist-test-slave-4wpb"
server_key: "a538cce9fcd7ddfba9d18616176c20c9"
server_key_iv: "42c15737db66ac419852a3bfe13a6f49"
server_key_version: "encryptionkey@0"
I20250903 14:26:53.899662 23320 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20250903 14:26:53.901316 23335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:53.902109 23320 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:53.902215 23320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "f6cfe423f36f406fa6593b9aa88f7018"
format_stamp: "Formatted at 2025-09-03 14:26:53 on dist-test-slave-4wpb"
server_key: "a538cce9fcd7ddfba9d18616176c20c9"
server_key_iv: "42c15737db66ac419852a3bfe13a6f49"
server_key_version: "encryptionkey@0"
I20250903 14:26:53.902280 23320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:53.916056 23320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:53.918146 23320 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:53.918295 23320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:53.924042 23320 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42825
I20250903 14:26:53.924070 23387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:42825 every 8 connection(s)
I20250903 14:26:53.924894 23320 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:53.927145 23388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:53.929643 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23320
I20250903 14:26:53.929770 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:53.929996 19718 external_mini_cluster.cc:1468] Setting key 8f12e6c3d6fdf7d183fbac3c3d460ae3
I20250903 14:26:53.932036 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018: Bootstrap starting.
I20250903 14:26:53.933885 23388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:53.934544 23388 log.cc:826] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:53.936009 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018: No bootstrap required, opened a new log
Sep 03 14:26:53 dist-test-slave-4wpb krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:53.938369 23388 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } }
I20250903 14:26:53.938510 23388 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:53.938573 23388 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cfe423f36f406fa6593b9aa88f7018, State: Initialized, Role: FOLLOWER
I20250903 14:26:53.938961 23388 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } }
I20250903 14:26:53.939096 23388 raft_consensus.cc:397] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:53.939173 23388 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:53.939280 23388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:53.940052 23388 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } }
I20250903 14:26:53.940336 23388 leader_election.cc:304] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6cfe423f36f406fa6593b9aa88f7018; no voters:
I20250903 14:26:53.940514 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:53.931042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.19.65.190:42825)
0903 14:26:53.931358 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:53.931365 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:53.931391 (+ 26us) server_negotiation.cc:408] Connection header received
0903 14:26:53.931798 (+ 407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:53.931810 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:53.932039 (+ 229us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:53.932300 (+ 261us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:53.933126 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:53.933896 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:53.934535 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:53.934782 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:53.936967 (+ 2185us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:53.936982 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:53.936990 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:53.937015 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:53.938457 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:53.938871 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:53.938875 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:53.938878 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:53.938933 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:53.939138 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:53.939140 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:53.939141 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:53.939455 (+ 314us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:53.939578 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:53.939798 (+ 220us) server_negotiation.cc:300] Negotiation successful
0903 14:26:53.939979 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
I20250903 14:26:53.940553 23388 leader_election.cc:290] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:53.940615 23393 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:53.940780 23393 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [term 1 LEADER]: Becoming Leader. State: Replica: f6cfe423f36f406fa6593b9aa88f7018, State: Running, Role: LEADER
I20250903 14:26:53.941064 23393 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } }
I20250903 14:26:53.941483 23388 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:53.942814 23394 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cfe423f36f406fa6593b9aa88f7018. Latest consensus state: current_term: 1 leader_uuid: "f6cfe423f36f406fa6593b9aa88f7018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } } }
I20250903 14:26:53.942909 23394 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:53.943212 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6cfe423f36f406fa6593b9aa88f7018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cfe423f36f406fa6593b9aa88f7018" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42825 } } }
I20250903 14:26:53.943276 23403 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:53.943300 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:53.945417 23403 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:53.949391 23403 catalog_manager.cc:1349] Generated new cluster ID: dcfbdafea55e4977af84cd51d1eed5e4
I20250903 14:26:53.949453 23403 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:53.959744 23403 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:53.960598 23403 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:53.969919 23403 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f6cfe423f36f406fa6593b9aa88f7018: Generated new TSK 0
I20250903 14:26:53.970386 23403 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:54.013828 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42825
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:54.100596 23416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:54.100773 23416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:54.100854 23416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:54.103577 23416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:54.103632 23416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:54.103742 23416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:54.107472 23416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:54.108235 23416 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:54.108856 23416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:54.113794 23424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:54.113842 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:54.113885 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:54.113981 23416 server_base.cc:1047] running on GCE node
I20250903 14:26:54.114341 23416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:54.114765 23416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:54.115901 23416 hybrid_clock.cc:648] HybridClock initialized: now 1756909614115890 us; error 28 us; skew 500 ppm
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:54.117936 23416 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:54.118701 23416 webserver.cc:480] Webserver started at http://127.19.65.129:42489/ using document root <none> and password file <none>
I20250903 14:26:54.119112 23416 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:54.119164 23416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:54.119310 23416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:54.120661 23416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "b03402ddc11f4e2f9b0d1408db8e8919"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "4e6aa08b9422e647a36856eba72a0ba9"
server_key_iv: "ab253445854fda9d4af48d10337868c1"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.121049 23416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "b03402ddc11f4e2f9b0d1408db8e8919"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "4e6aa08b9422e647a36856eba72a0ba9"
server_key_iv: "ab253445854fda9d4af48d10337868c1"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.123744 23416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:26:54.125442 23431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.126223 23416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:26:54.126312 23416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "b03402ddc11f4e2f9b0d1408db8e8919"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "4e6aa08b9422e647a36856eba72a0ba9"
server_key_iv: "ab253445854fda9d4af48d10337868c1"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.126381 23416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:54.136212 23416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:54.138211 23416 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:54.138345 23416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:54.138739 23416 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:54.139371 23416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:54.139418 23416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.139467 23416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:54.139492 23416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.147205 23416 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:35877
I20250903 14:26:54.147228 23544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:35877 every 8 connection(s)
I20250903 14:26:54.147930 23416 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:54.157586 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.149244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36211 (local address 127.19.65.190:42825)
0903 14:26:54.149361 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:54.149364 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:54.149877 (+ 513us) server_negotiation.cc:408] Connection header received
0903 14:26:54.150546 (+ 669us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:54.150550 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:54.150606 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:54.150670 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:54.151957 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.152398 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.152930 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.153117 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.154938 (+ 1821us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:54.154954 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:54.154956 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:54.154978 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:54.156095 (+ 1117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.156489 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.156492 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.156493 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.156531 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.156750 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.156752 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.156754 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.156870 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:54.156947 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:54.157316 (+ 369us) server_negotiation.cc:300] Negotiation successful
0903 14:26:54.157422 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:26:54.157902 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23416
I20250903 14:26:54.158001 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:54.158094 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.149416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42825 (local address 127.19.65.129:36211)
0903 14:26:54.149761 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:54.149789 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:54.150367 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:54.150809 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:54.150815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:54.151092 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:54.151816 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.151828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.152501 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.152504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:54.152826 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.152833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.152984 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.153543 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:54.153564 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:54.154809 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:54.156195 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.156199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.156207 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.156404 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.156602 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.156606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.156608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.156674 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.156957 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:54.156963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:54.157120 (+ 157us) client_negotiation.cc:769] Sending connection context
0903 14:26:54.157272 (+ 152us) client_negotiation.cc:240] Negotiation successful
0903 14:26:54.157493 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20250903 14:26:54.158244 19718 external_mini_cluster.cc:1468] Setting key 64408aa1be08cc6d89427cc18d002183
I20250903 14:26:54.158989 23545 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42825
I20250903 14:26:54.159183 23545 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:54.159637 23545 heartbeater.cc:507] Master 127.19.65.190:42825 requested a full tablet report, sending...
I20250903 14:26:54.160782 23347 ts_manager.cc:194] Registered new tserver with Master: b03402ddc11f4e2f9b0d1408db8e8919 (127.19.65.129:35877)
I20250903 14:26:54.161772 23347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:36211
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:54.194298 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42825
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:54.280728 23552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:54.280874 23552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:54.280925 23552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:54.283550 23552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:54.283598 23552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:54.283710 23552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:54.287081 23552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:54.287911 23552 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:54.288563 23552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:54.293766 23560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:54.293886 23557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:54.293843 23558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:54.294039 23552 server_base.cc:1047] running on GCE node
I20250903 14:26:54.294332 23552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:54.294754 23552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:54.295889 23552 hybrid_clock.cc:648] HybridClock initialized: now 1756909614295871 us; error 32 us; skew 500 ppm
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:54.298012 23552 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:54.298807 23552 webserver.cc:480] Webserver started at http://127.19.65.130:37443/ using document root <none> and password file <none>
I20250903 14:26:54.299214 23552 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:54.299261 23552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:54.299410 23552 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:54.300753 23552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "af8b446ebb77490b802608fb70a6fd6e"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "c71c47e1ae71b3ec410d77220638183d"
server_key_iv: "ce318b88fb29905f5cad0378bf1528d3"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.301151 23552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "af8b446ebb77490b802608fb70a6fd6e"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "c71c47e1ae71b3ec410d77220638183d"
server_key_iv: "ce318b88fb29905f5cad0378bf1528d3"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.303897 23552 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:26:54.305632 23567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.306361 23552 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:26:54.306454 23552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "af8b446ebb77490b802608fb70a6fd6e"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "c71c47e1ae71b3ec410d77220638183d"
server_key_iv: "ce318b88fb29905f5cad0378bf1528d3"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.306522 23552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:54.316524 23552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:54.318642 23552 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:54.318783 23552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:54.319176 23552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:54.319813 23552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:54.319854 23552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.319904 23552 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:54.319936 23552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.327289 23552 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41573
I20250903 14:26:54.327315 23680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:41573 every 8 connection(s)
I20250903 14:26:54.328040 23552 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:54.337867 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.329353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:56701 (local address 127.19.65.190:42825)
0903 14:26:54.329513 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:54.329518 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:54.330064 (+ 546us) server_negotiation.cc:408] Connection header received
0903 14:26:54.330776 (+ 712us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:54.330779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:54.330835 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:54.330904 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:54.332191 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.332616 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.333191 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.333378 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.335178 (+ 1800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:54.335193 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:54.335195 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:54.335217 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:54.336230 (+ 1013us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.336612 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.336615 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.336616 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.336654 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.336870 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.336872 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.336874 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.336989 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:54.337065 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:54.337612 (+ 547us) server_negotiation.cc:300] Negotiation successful
0903 14:26:54.337723 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250903 14:26:54.338207 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23552
I20250903 14:26:54.338312 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:54.338301 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.329571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42825 (local address 127.19.65.130:56701)
0903 14:26:54.329927 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:54.329967 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:54.330587 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:54.331045 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:54.331052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:54.331334 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:54.332044 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.332055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.332718 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.332721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:54.333037 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.333043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.333242 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.333785 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:54.333806 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:54.335038 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:54.336327 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.336332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.336340 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.336528 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.336725 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.336730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.336732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.336796 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.337072 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:54.337080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:54.337353 (+ 273us) client_negotiation.cc:769] Sending connection context
0903 14:26:54.337563 (+ 210us) client_negotiation.cc:240] Negotiation successful
0903 14:26:54.337767 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20250903 14:26:54.338555 19718 external_mini_cluster.cc:1468] Setting key ed366dcb845b99c66b275d082c123217
I20250903 14:26:54.339269 23681 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42825
I20250903 14:26:54.339474 23681 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:54.339911 23681 heartbeater.cc:507] Master 127.19.65.190:42825 requested a full tablet report, sending...
I20250903 14:26:54.340775 23347 ts_manager.cc:194] Registered new tserver with Master: af8b446ebb77490b802608fb70a6fd6e (127.19.65.130:41573)
I20250903 14:26:54.341384 23347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:56701
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:54.375937 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42825
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:54.462473 23688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:54.462615 23688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:54.462659 23688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:54.465353 23688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:54.465395 23688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:54.465456 23688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:54.468699 23688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:54.469384 23688 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:54.470083 23688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:54.475284 23696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:54.475284 23693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:54.475373 23688 server_base.cc:1047] running on GCE node
W20250903 14:26:54.475306 23694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:54.475710 23688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:54.476085 23688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:54.477216 23688 hybrid_clock.cc:648] HybridClock initialized: now 1756909614477207 us; error 24 us; skew 500 ppm
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:54.479485 23688 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:54.480301 23688 webserver.cc:480] Webserver started at http://127.19.65.131:38019/ using document root <none> and password file <none>
I20250903 14:26:54.480752 23688 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:54.480796 23688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:54.480978 23688 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:54.482362 23688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "67e268a8d830484b947dee84448ca8d1"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "49254b40e31c5f9e488c18a396996104"
server_key_iv: "7a90707d85f81cda4eb21ee01e1498e0"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.482736 23688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "67e268a8d830484b947dee84448ca8d1"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "49254b40e31c5f9e488c18a396996104"
server_key_iv: "7a90707d85f81cda4eb21ee01e1498e0"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.485455 23688 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 14:26:54.487149 23703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.487900 23688 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:26:54.487986 23688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "67e268a8d830484b947dee84448ca8d1"
format_stamp: "Formatted at 2025-09-03 14:26:54 on dist-test-slave-4wpb"
server_key: "49254b40e31c5f9e488c18a396996104"
server_key_iv: "7a90707d85f81cda4eb21ee01e1498e0"
server_key_version: "encryptionkey@0"
I20250903 14:26:54.488059 23688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:54.501420 23688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:54.508437 23688 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:54.508572 23688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:54.508977 23688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:54.509681 23688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:54.509723 23688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.509758 23688 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:54.509788 23688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:54.517305 23688 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:34215
I20250903 14:26:54.517349 23816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:34215 every 8 connection(s)
I20250903 14:26:54.518003 23688 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:54.519645 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23688
I20250903 14:26:54.519752 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:54.519945 19718 external_mini_cluster.cc:1468] Setting key 630f616ac93675b462a63289bcb34b2e
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:54.527566 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.519339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40731 (local address 127.19.65.190:42825)
0903 14:26:54.519467 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:54.519471 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:54.520014 (+ 543us) server_negotiation.cc:408] Connection header received
0903 14:26:54.520720 (+ 706us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:54.520723 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:54.520778 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:54.520840 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:54.522169 (+ 1329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.522591 (+ 422us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.523135 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.523317 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.525017 (+ 1700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:54.525029 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:54.525031 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:54.525051 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:54.526084 (+ 1033us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.526477 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.526480 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.526481 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.526519 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.526735 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.526738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.526740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.526873 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:54.526953 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:54.527334 (+ 381us) server_negotiation.cc:300] Negotiation successful
0903 14:26:54.527435 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:26:54.528076 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.519560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42825 (local address 127.19.65.131:40731)
0903 14:26:54.519908 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:54.519942 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:54.520558 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:54.520969 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:54.520975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:54.521256 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:54.522037 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.522049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.522692 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.522694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:54.523012 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:54.523017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.523195 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.523690 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:54.523710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:54.524899 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:54.526195 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.526199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.526207 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.526395 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.526594 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:54.526597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:54.526598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:54.526662 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:54.526962 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:54.526969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:54.527136 (+ 167us) client_negotiation.cc:769] Sending connection context
0903 14:26:54.527292 (+ 156us) client_negotiation.cc:240] Negotiation successful
0903 14:26:54.527492 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20250903 14:26:54.528868 23817 heartbeater.cc:344] Connected to a master server at 127.19.65.190:42825
I20250903 14:26:54.529044 23817 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:54.529449 23817 heartbeater.cc:507] Master 127.19.65.190:42825 requested a full tablet report, sending...
I20250903 14:26:54.530308 23347 ts_manager.cc:194] Registered new tserver with Master: 67e268a8d830484b947dee84448ca8d1 (127.19.65.131:34215)
I20250903 14:26:54.530779 23347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:40731
I20250903 14:26:54.532806 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:54.552507 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:54.546429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.19.65.190:42825)
0903 14:26:54.546548 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:54.546552 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:54.546667 (+ 115us) server_negotiation.cc:408] Connection header received
0903 14:26:54.546770 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:54.546773 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:54.546808 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:54.546865 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:54.547538 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.547921 (+ 383us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:54.548446 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:54.548624 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:54.550329 (+ 1705us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:54.550341 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:54.550343 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:54.550361 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:54.551345 (+ 984us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.551653 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.551656 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.551657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.551690 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:54.551866 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:54.551869 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:54.551871 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:54.551997 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:54.552101 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:54.552280 (+ 179us) server_negotiation.cc:300] Negotiation successful
0903 14:26:54.552370 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20250903 14:26:54.552778 23347 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45168
I20250903 14:26:54.553508 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23416
I20250903 14:26:54.558688 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23552
I20250903 14:26:54.563395 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23688
I20250903 14:26:54.567787 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23320
2025-09-03T14:26:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5019 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23829](info): set up 2 sockets
Sep 03 14:26:54 dist-test-slave-4wpb krb5kdc[23829](info): commencing operation
krb5kdc: starting...
W20250903 14:26:56.409686 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:56 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:56.528304 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44731
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:56.616482 23845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:56.616632 23845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:56.616675 23845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:56.619241 23845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:56.619287 23845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:56.619305 23845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:56.619320 23845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:56.619334 23845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:56.622604 23845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:56.623368 23845 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:56.624032 23845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:56.628396 23853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:56.628402 23850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:56.628469 23845 server_base.cc:1047] running on GCE node
W20250903 14:26:56.628402 23851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:56.628988 23845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:56.629719 23845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:56.630851 23845 hybrid_clock.cc:648] HybridClock initialized: now 1756909616630843 us; error 21 us; skew 500 ppm
Sep 03 14:26:56 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909616, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:56.632929 23845 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:56.633849 23845 webserver.cc:480] Webserver started at http://127.19.65.190:43205/ using document root <none> and password file <none>
I20250903 14:26:56.634320 23845 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:56.634387 23845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:56.634528 23845 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:56.635849 23845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "d929a703877f4afb8db6e3a93f004fc8"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "333ad0872713068a82ba8dddb39396af"
server_key_iv: "f9011673ac2b7d19fc52cbf0ecce79a3"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.636265 23845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "d929a703877f4afb8db6e3a93f004fc8"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "333ad0872713068a82ba8dddb39396af"
server_key_iv: "f9011673ac2b7d19fc52cbf0ecce79a3"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.639034 23845 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 14:26:56.640686 23860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:56.641426 23845 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:26:56.641538 23845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "d929a703877f4afb8db6e3a93f004fc8"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "333ad0872713068a82ba8dddb39396af"
server_key_iv: "f9011673ac2b7d19fc52cbf0ecce79a3"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.641607 23845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:56.666996 23845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:56.669013 23845 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:56.669144 23845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:56.674747 23912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:44731 every 8 connection(s)
I20250903 14:26:56.674748 23845 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44731
I20250903 14:26:56.675509 23845 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:56.677472 23913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:56.681620 23913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: Bootstrap starting.
I20250903 14:26:56.682487 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23845
I20250903 14:26:56.682587 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:56.682777 19718 external_mini_cluster.cc:1468] Setting key 1910faad0d392ca0a890a7f799b9bc85
I20250903 14:26:56.683647 23913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:56.684448 23913 log.cc:826] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: Log is configured to *not* fsync() on all Append() calls
I20250903 14:26:56.685973 23913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: No bootstrap required, opened a new log
I20250903 14:26:56.688468 23913 raft_consensus.cc:357] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } }
I20250903 14:26:56.688602 23913 raft_consensus.cc:383] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:56.688632 23913 raft_consensus.cc:738] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d929a703877f4afb8db6e3a93f004fc8, State: Initialized, Role: FOLLOWER
I20250903 14:26:56.688956 23913 consensus_queue.cc:260] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } }
I20250903 14:26:56.689056 23913 raft_consensus.cc:397] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:56.689106 23913 raft_consensus.cc:491] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 03 14:26:56 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:56.689188 23913 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:56.689987 23913 raft_consensus.cc:513] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } }
I20250903 14:26:56.690315 23913 leader_election.cc:304] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d929a703877f4afb8db6e3a93f004fc8; no voters:
I20250903 14:26:56.690570 23913 leader_election.cc:290] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:56.690851 23918 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:56.691079 23918 raft_consensus.cc:695] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [term 1 LEADER]: Becoming Leader. State: Replica: d929a703877f4afb8db6e3a93f004fc8, State: Running, Role: LEADER
I20250903 14:26:56.691360 23918 consensus_queue.cc:237] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } }
I20250903 14:26:56.691428 23913 sys_catalog.cc:564] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:56.692709 23919 sys_catalog.cc:455] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d929a703877f4afb8db6e3a93f004fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } } }
I20250903 14:26:56.692799 23919 sys_catalog.cc:458] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:56.693159 23920 sys_catalog.cc:455] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d929a703877f4afb8db6e3a93f004fc8. Latest consensus state: current_term: 1 leader_uuid: "d929a703877f4afb8db6e3a93f004fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d929a703877f4afb8db6e3a93f004fc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44731 } } }
I20250903 14:26:56.693325 23920 sys_catalog.cc:458] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:56.694185 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:56.683626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.19.65.190:44731)
0903 14:26:56.684020 (+ 394us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:56.684029 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:56.684064 (+ 35us) server_negotiation.cc:408] Connection header received
0903 14:26:56.684652 (+ 588us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:56.684673 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:56.684920 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:56.685170 (+ 250us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:56.685827 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.686822 (+ 995us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:56.687505 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.687780 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:56.689650 (+ 1870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:56.689668 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:56.689679 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:56.689712 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:56.691534 (+ 1822us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:56.691991 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:56.691995 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:56.691999 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:56.692072 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:56.692324 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:56.692326 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:56.692328 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:56.692587 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:56.692733 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:56.693317 (+ 584us) server_negotiation.cc:300] Negotiation successful
0903 14:26:56.693562 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
W20250903 14:26:56.695735 23933 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:26:56.695801 23933 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:26:56.696066 23927 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:56.696688 23927 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:56.700696 23927 catalog_manager.cc:1349] Generated new cluster ID: 3157bb5733f94764bc249794a9047afb
I20250903 14:26:56.700750 23927 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:56.713424 23927 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:56.714462 23927 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:56.724411 23927 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d929a703877f4afb8db6e3a93f004fc8: Generated new TSK 0
I20250903 14:26:56.724963 23927 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:56.767953 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44731
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:56.876114 23941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:56.876272 23941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:56.876322 23941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:56.878964 23941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:56.879019 23941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:56.879082 23941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:56.882486 23941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:56.883280 23941 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:56.884013 23941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:56.889233 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:56.889261 23949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:56.889241 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:56.889449 23941 server_base.cc:1047] running on GCE node
I20250903 14:26:56.889771 23941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:56.890240 23941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:56.891378 23941 hybrid_clock.cc:648] HybridClock initialized: now 1756909616891367 us; error 28 us; skew 500 ppm
Sep 03 14:26:56 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909616, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:56.893869 23941 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:56.894737 23941 webserver.cc:480] Webserver started at http://127.19.65.129:41535/ using document root <none> and password file <none>
I20250903 14:26:56.895174 23941 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:56.895222 23941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:56.895412 23941 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:56.896818 23941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "307ba06a1bb84fabb16c0644d692da48"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "812d0699ddbe8b3d3ccb134703a4b071"
server_key_iv: "162b744559b869cf0d3afe4803d2388d"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.897228 23941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "307ba06a1bb84fabb16c0644d692da48"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "812d0699ddbe8b3d3ccb134703a4b071"
server_key_iv: "162b744559b869cf0d3afe4803d2388d"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.899971 23941 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:26:56.901683 23956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:56.902479 23941 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:26:56.902566 23941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "307ba06a1bb84fabb16c0644d692da48"
format_stamp: "Formatted at 2025-09-03 14:26:56 on dist-test-slave-4wpb"
server_key: "812d0699ddbe8b3d3ccb134703a4b071"
server_key_iv: "162b744559b869cf0d3afe4803d2388d"
server_key_version: "encryptionkey@0"
I20250903 14:26:56.902639 23941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:56.913381 23941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:56.915482 23941 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:56.915612 23941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:56.915999 23941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:56.916678 23941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:56.916720 23941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:56.916769 23941 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:56.916800 23941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:56.924110 23941 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:37139
I20250903 14:26:56.924136 24069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:37139 every 8 connection(s)
I20250903 14:26:56.924885 23941 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:56 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909616, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:56.932379 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 23941
I20250903 14:26:56.932484 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:56.932682 19718 external_mini_cluster.cc:1468] Setting key ab072cb3f794a11716e1396d298e9a5b
I20250903 14:26:56.934881 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:56.926221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:46029 (local address 127.19.65.190:44731)
0903 14:26:56.926338 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:56.926342 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:56.926853 (+ 511us) server_negotiation.cc:408] Connection header received
0903 14:26:56.927543 (+ 690us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:56.927547 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:56.927604 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:56.927670 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:56.928665 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.929094 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:56.930046 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.930207 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:56.931787 (+ 1580us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:56.931801 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:56.931803 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:56.931823 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:56.933009 (+ 1186us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:56.933629 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:56.933632 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:56.933634 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:56.933681 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:56.933979 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:56.933982 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:56.933983 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:56.934119 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:56.934194 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:56.934699 (+ 505us) server_negotiation.cc:300] Negotiation successful
0903 14:26:56.934777 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:26:56.935596 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:56.926410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44731 (local address 127.19.65.129:46029)
0903 14:26:56.926745 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:56.926774 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:56.927373 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:56.927807 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:56.927813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:56.928110 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:56.928540 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:56.928550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.929193 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:56.929195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:56.929937 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:56.929945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:56.930099 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:56.930559 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:56.930578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:56.931667 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:56.933271 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:56.933276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:56.933283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:56.933524 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:56.933777 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:56.933780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:56.933783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:56.933878 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:56.934236 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:56.934243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:56.934482 (+ 239us) client_negotiation.cc:769] Sending connection context
0903 14:26:56.934673 (+ 191us) client_negotiation.cc:240] Negotiation successful
0903 14:26:56.934892 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20250903 14:26:56.936558 24070 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44731
I20250903 14:26:56.936772 24070 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:56.937318 24070 heartbeater.cc:507] Master 127.19.65.190:44731 requested a full tablet report, sending...
I20250903 14:26:56.938762 23877 ts_manager.cc:194] Registered new tserver with Master: 307ba06a1bb84fabb16c0644d692da48 (127.19.65.129:37139)
I20250903 14:26:56.940135 23877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:46029
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:56.970197 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44731
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:57.055905 24077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:57.056046 24077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:57.056089 24077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:57.058833 24077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:57.058876 24077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:57.058938 24077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:57.062165 24077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:57.062856 24077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:57.063550 24077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:57.068558 24083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:57.068540 24085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:57.068571 24082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:57.068718 24077 server_base.cc:1047] running on GCE node
I20250903 14:26:57.069020 24077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:57.069396 24077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:57.070528 24077 hybrid_clock.cc:648] HybridClock initialized: now 1756909617070515 us; error 28 us; skew 500 ppm
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:57.072680 24077 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:57.073549 24077 webserver.cc:480] Webserver started at http://127.19.65.130:34633/ using document root <none> and password file <none>
I20250903 14:26:57.074021 24077 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:57.074067 24077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:57.074223 24077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:57.075559 24077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "37b8cc869f374488983af8572e3b0f3f"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "9981524139589d5fe7d7d21735863812"
server_key_iv: "846ce88c72826eb8cc90303599c8d0fd"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.075939 24077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "37b8cc869f374488983af8572e3b0f3f"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "9981524139589d5fe7d7d21735863812"
server_key_iv: "846ce88c72826eb8cc90303599c8d0fd"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.078500 24077 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:26:57.080092 24092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.080837 24077 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 14:26:57.080921 24077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "37b8cc869f374488983af8572e3b0f3f"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "9981524139589d5fe7d7d21735863812"
server_key_iv: "846ce88c72826eb8cc90303599c8d0fd"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.081000 24077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:57.114964 24077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:57.116921 24077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:57.117066 24077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:57.117460 24077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:57.118108 24077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:57.118149 24077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.118197 24077 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:57.118223 24077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.125391 24077 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43603
I20250903 14:26:57.125413 24205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:43603 every 8 connection(s)
I20250903 14:26:57.126113 24077 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:57.134336 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24077
I20250903 14:26:57.134447 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:57.134656 19718 external_mini_cluster.cc:1468] Setting key b3ab786b1372b775cdfdf83d1fac1238
I20250903 14:26:57.137639 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.127584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:32783 (local address 127.19.65.190:44731)
0903 14:26:57.127718 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:57.127722 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:57.128238 (+ 516us) server_negotiation.cc:408] Connection header received
0903 14:26:57.128975 (+ 737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:57.128979 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:57.129035 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:57.129102 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:57.130111 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.130537 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.131468 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.131632 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.133341 (+ 1709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:57.133358 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:57.133360 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:57.133386 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:57.134855 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.135266 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.135269 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.135271 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.135318 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.135547 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.135550 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.135551 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.135677 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:57.135756 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:57.137379 (+ 1623us) server_negotiation.cc:300] Negotiation successful
0903 14:26:57.137506 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 14:26:57.138170 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.127761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44731 (local address 127.19.65.130:32783)
0903 14:26:57.128118 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:57.128149 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:57.128795 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:57.129244 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:57.129250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:57.129570 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:57.129983 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:57.129993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.130641 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.130643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:57.131359 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:57.131367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.131523 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.132050 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:57.132068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:57.133203 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:57.134957 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:57.134962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:57.134976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:57.135174 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:57.135403 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:57.135408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:57.135410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:57.135473 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:57.137020 (+ 1547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:57.137024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:57.137181 (+ 157us) client_negotiation.cc:769] Sending connection context
0903 14:26:57.137333 (+ 152us) client_negotiation.cc:240] Negotiation successful
0903 14:26:57.137548 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20250903 14:26:57.139016 24206 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44731
I20250903 14:26:57.139192 24206 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:57.139559 24206 heartbeater.cc:507] Master 127.19.65.190:44731 requested a full tablet report, sending...
I20250903 14:26:57.140360 23877 ts_manager.cc:194] Registered new tserver with Master: 37b8cc869f374488983af8572e3b0f3f (127.19.65.130:43603)
I20250903 14:26:57.140805 23877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:32783
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:57.173394 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44731
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:57.259549 24213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:57.259689 24213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:57.259732 24213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:57.262302 24213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:57.262346 24213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:57.262427 24213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:57.265893 24213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:57.266600 24213 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:57.267225 24213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:57.271947 24218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:57.271951 24219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:57.271957 24221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:57.272238 24213 server_base.cc:1047] running on GCE node
I20250903 14:26:57.272521 24213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:57.272936 24213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:57.274065 24213 hybrid_clock.cc:648] HybridClock initialized: now 1756909617274054 us; error 26 us; skew 500 ppm
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:57.276247 24213 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:57.277016 24213 webserver.cc:480] Webserver started at http://127.19.65.131:43245/ using document root <none> and password file <none>
I20250903 14:26:57.277432 24213 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:57.277496 24213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:57.277640 24213 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:57.278923 24213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "c887a50ab4aa44b59a7905f3e6d146fc"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "1e86541525ee9e4c9a4655d6bdea3055"
server_key_iv: "2b3e82ac583ec985716aee1af9c57d98"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.279302 24213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "c887a50ab4aa44b59a7905f3e6d146fc"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "1e86541525ee9e4c9a4655d6bdea3055"
server_key_iv: "2b3e82ac583ec985716aee1af9c57d98"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.281876 24213 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:26:57.283634 24228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.284475 24213 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:57.284562 24213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "c887a50ab4aa44b59a7905f3e6d146fc"
format_stamp: "Formatted at 2025-09-03 14:26:57 on dist-test-slave-4wpb"
server_key: "1e86541525ee9e4c9a4655d6bdea3055"
server_key_iv: "2b3e82ac583ec985716aee1af9c57d98"
server_key_version: "encryptionkey@0"
I20250903 14:26:57.284632 24213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:57.295439 24213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:57.297582 24213 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:57.297711 24213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:57.298094 24213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:57.298700 24213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:57.298743 24213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.298789 24213 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:57.298815 24213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:57.306298 24213 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:43647
I20250903 14:26:57.306335 24341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:43647 every 8 connection(s)
I20250903 14:26:57.307080 24213 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:57.316493 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.308434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:45167 (local address 127.19.65.190:44731)
0903 14:26:57.308553 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:57.308556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:57.309018 (+ 462us) server_negotiation.cc:408] Connection header received
0903 14:26:57.309624 (+ 606us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:57.309626 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:57.309671 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:57.309735 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:57.310712 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.311126 (+ 414us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.311978 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.312094 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.313898 (+ 1804us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:57.313913 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:57.313916 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:57.313950 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:57.315204 (+ 1254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.315638 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.315640 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.315642 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.315672 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.315841 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.315842 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.315843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.315949 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:57.316017 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:57.316320 (+ 303us) server_negotiation.cc:300] Negotiation successful
0903 14:26:57.316392 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 14:26:57.316926 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.308583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44731 (local address 127.19.65.131:45167)
0903 14:26:57.308894 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:57.308921 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:57.309460 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:57.309900 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:57.309907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:57.310181 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:57.310603 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:57.310614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.311203 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.311205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:57.311896 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:57.311903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.312026 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.312492 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:57.312511 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:57.313771 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:57.315378 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:57.315382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:57.315388 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:57.315565 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:57.315732 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:57.315734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:57.315735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:57.315788 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:57.316016 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:57.316019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:57.316161 (+ 142us) client_negotiation.cc:769] Sending connection context
0903 14:26:57.316294 (+ 133us) client_negotiation.cc:240] Negotiation successful
0903 14:26:57.316442 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20250903 14:26:57.317142 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24213
I20250903 14:26:57.317234 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:26:57.317452 19718 external_mini_cluster.cc:1468] Setting key 34ac7e3f0fc4b466b06c7ffc97c01a7f
I20250903 14:26:57.317847 24342 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44731
I20250903 14:26:57.318046 24342 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:57.318492 24342 heartbeater.cc:507] Master 127.19.65.190:44731 requested a full tablet report, sending...
I20250903 14:26:57.319274 23877 ts_manager.cc:194] Registered new tserver with Master: c887a50ab4aa44b59a7905f3e6d146fc (127.19.65.131:43647)
I20250903 14:26:57.319725 23877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:45167
I20250903 14:26:57.330402 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:26:57.349762 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.342700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.19.65.129:37139)
0903 14:26:57.342990 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:57.342995 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:57.343013 (+ 18us) server_negotiation.cc:408] Connection header received
0903 14:26:57.343051 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:57.343054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:57.343183 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:57.343301 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:57.343965 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.344624 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.345297 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.345465 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.347322 (+ 1857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:57.347339 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:57.347346 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:57.347365 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:57.348580 (+ 1215us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.348895 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.348899 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.348900 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.348936 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.349097 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.349098 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.349099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.349216 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:57.349341 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:57.349454 (+ 113us) server_negotiation.cc:300] Negotiation successful
0903 14:26:57.349580 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[23829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:57.358872 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:57.352129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.19.65.190:44731)
0903 14:26:57.352248 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:57.352252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:57.352474 (+ 222us) server_negotiation.cc:408] Connection header received
0903 14:26:57.352592 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:57.352595 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:57.352638 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:57.352704 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:57.353366 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.354011 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:57.354636 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:57.354825 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:57.356527 (+ 1702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:57.356539 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:57.356541 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:57.356559 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:57.357565 (+ 1006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.357864 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.357867 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.357869 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.357906 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:57.358110 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:57.358113 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:57.358114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:57.358244 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:57.358347 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:57.358646 (+ 299us) server_negotiation.cc:300] Negotiation successful
0903 14:26:57.358742 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20250903 14:26:57.359159 23877 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36072
I20250903 14:26:57.359633 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23941
I20250903 14:26:57.364805 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24077
I20250903 14:26:57.369369 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24213
I20250903 14:26:57.374025 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 23845
2025-09-03T14:26:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2805 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[24354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[24354](info): set up 2 sockets
Sep 03 14:26:57 dist-test-slave-4wpb krb5kdc[24354](info): commencing operation
krb5kdc: starting...
W20250903 14:26:59.125777 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.732s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:26:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:59.233976 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:38301
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:59.321396 24370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:59.321597 24370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:59.321672 24370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:59.324550 24370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:26:59.324599 24370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:59.324630 24370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:59.324666 24370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:26:59.324697 24370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:26:59.328081 24370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:59.328931 24370 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:59.329631 24370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:59.333958 24376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:59.333974 24378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:59.333958 24375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.334355 24370 server_base.cc:1047] running on GCE node
I20250903 14:26:59.334751 24370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:59.335464 24370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:59.336611 24370 hybrid_clock.cc:648] HybridClock initialized: now 1756909619336588 us; error 34 us; skew 500 ppm
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:59.339004 24370 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:26:59.339529 24370 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:59.339576 24370 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:59.339694 24370 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:59.341003 24370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "da58b4a599a0485e852d302eadf54c22"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "f93db37bbb4661f892c44c047d9749b7"
server_key_iv: "b3ebc29321c290808f3256706a82af99"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.341372 24370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "da58b4a599a0485e852d302eadf54c22"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "f93db37bbb4661f892c44c047d9749b7"
server_key_iv: "b3ebc29321c290808f3256706a82af99"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.344182 24370 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250903 14:26:59.346051 24384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.346943 24370 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:59.347051 24370 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "da58b4a599a0485e852d302eadf54c22"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "f93db37bbb4661f892c44c047d9749b7"
server_key_iv: "b3ebc29321c290808f3256706a82af99"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.347128 24370 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:59.375238 24370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:59.377367 24370 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:59.377548 24370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:59.383384 24436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:38301 every 8 connection(s)
I20250903 14:26:59.383387 24370 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:38301
I20250903 14:26:59.383914 24370 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:26:59.386019 24437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:26:59.388190 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24370
I20250903 14:26:59.388290 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:26:59.388523 19718 external_mini_cluster.cc:1468] Setting key d3179951916c4bd2b8ee662e57bd639d
I20250903 14:26:59.390604 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: Bootstrap starting.
I20250903 14:26:59.392639 24437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: Neither blocks nor log segments found. Creating new log.
I20250903 14:26:59.393414 24437 log.cc:826] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: Log is configured to *not* fsync() on all Append() calls
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:59.395156 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: No bootstrap required, opened a new log
I20250903 14:26:59.396955 24437 raft_consensus.cc:357] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } }
I20250903 14:26:59.397096 24437 raft_consensus.cc:383] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:26:59.397161 24437 raft_consensus.cc:738] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da58b4a599a0485e852d302eadf54c22, State: Initialized, Role: FOLLOWER
I20250903 14:26:59.397506 24437 consensus_queue.cc:260] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } }
I20250903 14:26:59.397616 24437 raft_consensus.cc:397] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:26:59.397673 24437 raft_consensus.cc:491] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:26:59.397768 24437 raft_consensus.cc:3058] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:26:59.398408 24437 raft_consensus.cc:513] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } }
I20250903 14:26:59.398681 24437 leader_election.cc:304] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da58b4a599a0485e852d302eadf54c22; no voters:
I20250903 14:26:59.398911 24437 leader_election.cc:290] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:26:59.398984 24442 raft_consensus.cc:2802] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:26:59.399175 24442 raft_consensus.cc:695] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [term 1 LEADER]: Becoming Leader. State: Replica: da58b4a599a0485e852d302eadf54c22, State: Running, Role: LEADER
I20250903 14:26:59.399243 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.389497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.19.65.190:38301)
0903 14:26:59.389804 (+ 307us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:59.389812 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:59.389856 (+ 44us) server_negotiation.cc:408] Connection header received
0903 14:26:59.390265 (+ 409us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:59.390277 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:59.390500 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:59.390747 (+ 247us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:59.391658 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.392326 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:59.393137 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.393342 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:59.395578 (+ 2236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:59.395592 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:59.395601 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:59.395625 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:59.396990 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.397414 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.397418 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.397421 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.397512 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.397728 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.397733 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.397734 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.398022 (+ 288us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:59.398189 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:59.398430 (+ 241us) server_negotiation.cc:300] Negotiation successful
0903 14:26:59.398656 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20250903 14:26:59.399550 24442 consensus_queue.cc:237] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } }
I20250903 14:26:59.399863 24437 sys_catalog.cc:564] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:26:59.400502 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da58b4a599a0485e852d302eadf54c22. Latest consensus state: current_term: 1 leader_uuid: "da58b4a599a0485e852d302eadf54c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } } }
I20250903 14:26:59.400647 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:59.400858 24443 sys_catalog.cc:455] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da58b4a599a0485e852d302eadf54c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da58b4a599a0485e852d302eadf54c22" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38301 } } }
I20250903 14:26:59.401015 24443 sys_catalog.cc:458] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22 [sys.catalog]: This master's current role is: LEADER
I20250903 14:26:59.401049 24451 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:26:59.403422 24451 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:26:59.407547 24451 catalog_manager.cc:1349] Generated new cluster ID: 40d7dbeeea694a69adc7f86bfab37424
I20250903 14:26:59.407600 24451 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:26:59.420156 24451 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:26:59.420857 24451 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:26:59.430553 24451 catalog_manager.cc:5955] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: Generated new TSK 0
I20250903 14:26:59.431036 24451 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:59.473249 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:38301
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:59.561837 24465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:59.561981 24465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:59.562029 24465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:59.564899 24465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:59.564944 24465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:59.565007 24465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:26:59.568305 24465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:59.569063 24465 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:59.569732 24465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:59.574731 24471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.574858 24465 server_base.cc:1047] running on GCE node
W20250903 14:26:59.574740 24470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:59.574786 24473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.575255 24465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:59.575680 24465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:59.576817 24465 hybrid_clock.cc:648] HybridClock initialized: now 1756909619576800 us; error 27 us; skew 500 ppm
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:59.579079 24465 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:26:59.579623 24465 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:59.579674 24465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:59.579815 24465 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:59.581125 24465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "293b1f0bee3e4938b5bdf1730c087483"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "e079b8889cda876b2b55c4d34b925701"
server_key_iv: "b2b70db01f902ab89bc7044d9a111d4a"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.581548 24465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "293b1f0bee3e4938b5bdf1730c087483"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "e079b8889cda876b2b55c4d34b925701"
server_key_iv: "b2b70db01f902ab89bc7044d9a111d4a"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.584254 24465 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:26:59.585929 24479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.586689 24465 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:59.586781 24465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "293b1f0bee3e4938b5bdf1730c087483"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "e079b8889cda876b2b55c4d34b925701"
server_key_iv: "b2b70db01f902ab89bc7044d9a111d4a"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.586846 24465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:59.606403 24465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:59.612520 24465 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:59.612658 24465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:59.613058 24465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:59.613808 24465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:59.613871 24465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.613926 24465 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:59.613960 24465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.621150 24465 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:43373
I20250903 14:26:59.621191 24592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:43373 every 8 connection(s)
I20250903 14:26:59.621562 24465 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:59.627663 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24465
I20250903 14:26:59.627765 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:26:59.627966 19718 external_mini_cluster.cc:1468] Setting key ca5392a2b6f0ad41017feef961b87d2b
I20250903 14:26:59.631250 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.622776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:59091 (local address 127.19.65.190:38301)
0903 14:26:59.622909 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:59.622914 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:59.623428 (+ 514us) server_negotiation.cc:408] Connection header received
0903 14:26:59.624113 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:59.624115 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:59.624162 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:59.624231 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:59.625452 (+ 1221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.625881 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:59.626371 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.626485 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:59.628098 (+ 1613us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:59.628124 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:59.628127 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:59.628151 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:59.629659 (+ 1508us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.630040 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.630046 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.630050 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.630094 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.630308 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.630312 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.630317 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.630480 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:59.630572 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:59.631062 (+ 490us) server_negotiation.cc:300] Negotiation successful
0903 14:26:59.631154 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 14:26:59.631904 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.622964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:38301 (local address 127.19.65.129:59091)
0903 14:26:59.623306 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:59.623337 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:59.623951 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:59.624346 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:59.624353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:59.624637 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:59.625355 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:59.625366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.625962 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:59.625965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:59.626291 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:59.626297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.626417 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:59.626788 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:59.626805 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:59.627982 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:59.629756 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:59.629760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:59.629767 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:59.629962 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:59.630176 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:59.630179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:59.630180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:59.630243 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:59.630587 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:59.630595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:59.630817 (+ 222us) client_negotiation.cc:769] Sending connection context
0903 14:26:59.631031 (+ 214us) client_negotiation.cc:240] Negotiation successful
0903 14:26:59.631251 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20250903 14:26:59.632818 24593 heartbeater.cc:344] Connected to a master server at 127.19.65.190:38301
I20250903 14:26:59.632997 24593 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:59.633369 24593 heartbeater.cc:507] Master 127.19.65.190:38301 requested a full tablet report, sending...
I20250903 14:26:59.634622 24401 ts_manager.cc:194] Registered new tserver with Master: 293b1f0bee3e4938b5bdf1730c087483 (127.19.65.129:43373)
I20250903 14:26:59.635535 24401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:59091
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:59.665391 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:38301
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:59.756146 24600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:59.756286 24600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:59.756335 24600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:59.758983 24600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:59.759027 24600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:59.759096 24600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:26:59.762379 24600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:59.763079 24600 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:59.763792 24600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:59.768676 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:59.768662 24608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.768748 24600 server_base.cc:1047] running on GCE node
W20250903 14:26:59.768662 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.769143 24600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:59.769593 24600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:59.770726 24600 hybrid_clock.cc:648] HybridClock initialized: now 1756909619770716 us; error 27 us; skew 500 ppm
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:59.772990 24600 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:26:59.773665 24600 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:59.773726 24600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:59.773913 24600 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:59.775353 24600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "9f06a16a69f247d19c91ff93e32bd714"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "3ca5043559fcebe790f8e4ba61cc1514"
server_key_iv: "17d6288d96e2520dd7373789b40ddfae"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.775727 24600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "9f06a16a69f247d19c91ff93e32bd714"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "3ca5043559fcebe790f8e4ba61cc1514"
server_key_iv: "17d6288d96e2520dd7373789b40ddfae"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.778255 24600 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:26:59.779949 24614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.780699 24600 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:59.780781 24600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "9f06a16a69f247d19c91ff93e32bd714"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "3ca5043559fcebe790f8e4ba61cc1514"
server_key_iv: "17d6288d96e2520dd7373789b40ddfae"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.780848 24600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:59.799430 24600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:59.801400 24600 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:59.801549 24600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:59.801962 24600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:59.802619 24600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:59.802661 24600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.802706 24600 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:59.802731 24600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.809993 24600 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:40671
I20250903 14:26:59.810040 24727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:40671 every 8 connection(s)
I20250903 14:26:59.810427 24600 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:26:59.819213 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24600
I20250903 14:26:59.819330 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:26:59.819543 19718 external_mini_cluster.cc:1468] Setting key 168f2e1f73d6c1cdbad2ce904be63f3e
I20250903 14:26:59.820398 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.811815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:44743 (local address 127.19.65.190:38301)
0903 14:26:59.811961 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:59.811965 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:59.812477 (+ 512us) server_negotiation.cc:408] Connection header received
0903 14:26:59.813097 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:59.813100 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:59.813142 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:59.813208 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:26:59.814648 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.815083 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:26:59.815626 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.815733 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:59.817497 (+ 1764us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:26:59.817520 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:26:59.817522 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:26:59.817544 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:26:59.818786 (+ 1242us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.819161 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.819164 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.819165 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.819196 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:26:59.819449 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:26:59.819451 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:26:59.819452 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:26:59.819567 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:26:59.819649 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0903 14:26:59.820143 (+ 494us) server_negotiation.cc:300] Negotiation successful
0903 14:26:59.820258 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 14:26:59.820881 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.812014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:38301 (local address 127.19.65.130:44743)
0903 14:26:59.812358 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:59.812386 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:59.812950 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:59.813338 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:59.813345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:59.813707 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:59.814537 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:59.814548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.815168 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:26:59.815171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:26:59.815550 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:26:59.815557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:26:59.815680 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:26:59.816145 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:26:59.816163 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:26:59.817352 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:26:59.818870 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:59.818873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:59.818879 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:59.819078 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:59.819261 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:26:59.819265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:26:59.819268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:26:59.819365 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:26:59.819682 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:26:59.819687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:26:59.819912 (+ 225us) client_negotiation.cc:769] Sending connection context
0903 14:26:59.820093 (+ 181us) client_negotiation.cc:240] Negotiation successful
0903 14:26:59.820305 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20250903 14:26:59.821717 24728 heartbeater.cc:344] Connected to a master server at 127.19.65.190:38301
I20250903 14:26:59.821926 24728 heartbeater.cc:461] Registering TS with master...
I20250903 14:26:59.822286 24728 heartbeater.cc:507] Master 127.19.65.190:38301 requested a full tablet report, sending...
I20250903 14:26:59.823060 24401 ts_manager.cc:194] Registered new tserver with Master: 9f06a16a69f247d19c91ff93e32bd714 (127.19.65.130:40671)
I20250903 14:26:59.823581 24401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:44743
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:26:59.855787 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:38301
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:26:59.942478 24735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:26:59.942620 24735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:26:59.942682 24735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:26:59.945545 24735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:26:59.945586 24735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:26:59.945680 24735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:26:59.949050 24735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:26:59.949793 24735 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:26:59.950450 24735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:26:59.955461 24740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:26:59.955471 24741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.955565 24735 server_base.cc:1047] running on GCE node
W20250903 14:26:59.955477 24743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:26:59.955960 24735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:26:59.956395 24735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:26:59.957525 24735 hybrid_clock.cc:648] HybridClock initialized: now 1756909619957507 us; error 31 us; skew 500 ppm
Sep 03 14:26:59 dist-test-slave-4wpb krb5kdc[24354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:26:59.959707 24735 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:26:59.960258 24735 fs_manager.cc:362] Metadata directory not provided
I20250903 14:26:59.960307 24735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:26:59.960430 24735 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:26:59.961758 24735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "ab87062cc0344496a09336cd4a112afd"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "9f9b22f0d384c1e8d87674e1c8647d70"
server_key_iv: "c9e768bb2ec341f6890ac92c74bfd013"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.962112 24735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "ab87062cc0344496a09336cd4a112afd"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "9f9b22f0d384c1e8d87674e1c8647d70"
server_key_iv: "c9e768bb2ec341f6890ac92c74bfd013"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.964717 24735 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:26:59.966457 24749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.967236 24735 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:26:59.967314 24735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "ab87062cc0344496a09336cd4a112afd"
format_stamp: "Formatted at 2025-09-03 14:26:59 on dist-test-slave-4wpb"
server_key: "9f9b22f0d384c1e8d87674e1c8647d70"
server_key_iv: "c9e768bb2ec341f6890ac92c74bfd013"
server_key_version: "encryptionkey@0"
I20250903 14:26:59.967393 24735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:26:59.984426 24735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:26:59.986804 24735 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:26:59.986955 24735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:26:59.987365 24735 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:26:59.988011 24735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:26:59.988050 24735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.988082 24735 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:26:59.988111 24735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:26:59.995718 24735 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41991
I20250903 14:26:59.995746 24862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:41991 every 8 connection(s)
I20250903 14:26:59.996109 24735 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:26:59.999965 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24735
I20250903 14:27:00.000067 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:00.000276 19718 external_mini_cluster.cc:1468] Setting key b5b108daf9aeebc2f25c5ecbe24e575a
Sep 03 14:27:00 dist-test-slave-4wpb krb5kdc[24354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909619, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:00.006225 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.997317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:51185 (local address 127.19.65.190:38301)
0903 14:26:59.997426 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:26:59.997430 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:26:59.997919 (+ 489us) server_negotiation.cc:408] Connection header received
0903 14:26:59.998610 (+ 691us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:26:59.998613 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:26:59.998667 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:26:59.998746 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.000175 (+ 1429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.000640 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.001308 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.001468 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.003241 (+ 1773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.003257 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.003259 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:00.003283 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.004488 (+ 1205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:00.004952 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:00.004955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:00.004956 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:00.005022 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:00.005271 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:00.005274 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:00.005277 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:00.005411 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.005525 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.005953 (+ 428us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.006066 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:27:00.006645 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:26:59.997498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:38301 (local address 127.19.65.131:51185)
0903 14:26:59.997813 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0903 14:26:59.997842 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0903 14:26:59.998448 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:26:59.998882 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:26:59.998890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:26:59.999184 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:26:59.999984 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.000002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.000740 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.000743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.001201 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.001209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.001359 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.001849 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:00.001879 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.003098 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.004583 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:00.004587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:00.004596 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:00.004800 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:00.005107 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:00.005110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:00.005114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:00.005187 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:00.005526 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.005532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.005717 (+ 185us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.005894 (+ 177us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.006124 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20250903 14:27:00.007442 24863 heartbeater.cc:344] Connected to a master server at 127.19.65.190:38301
I20250903 14:27:00.007627 24863 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:00.007987 24863 heartbeater.cc:507] Master 127.19.65.190:38301 requested a full tablet report, sending...
I20250903 14:27:00.008760 24401 ts_manager.cc:194] Registered new tserver with Master: ab87062cc0344496a09336cd4a112afd (127.19.65.131:41991)
I20250903 14:27:00.009212 24401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:51185
I20250903 14:27:00.013340 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:27:00.019657 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.014917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.19.65.190:38301)
0903 14:27:00.015048 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.015051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.015147 (+ 96us) server_negotiation.cc:408] Connection header received
0903 14:27:00.015223 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.015225 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.015256 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.015305 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.015890 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.016265 (+ 375us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.016734 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.016861 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.017709 (+ 848us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.017720 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.017722 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:00.017738 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.018720 (+ 982us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:00.019009 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:00.019011 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:00.019012 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:00.019042 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:00.019173 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:00.019175 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:00.019176 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:00.019284 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.019364 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.019467 (+ 103us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.019536 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250903 14:27:00.021674 24401 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:00.023660 24401 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:00.033537 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.029938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:40671 (local address 127.0.0.1:41418)
0903 14:27:00.030379 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.030410 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.030520 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.031285 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.031292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.031316 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:00.031540 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.031547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032458 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.032462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.033077 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.033084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.033171 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.033208 (+ 37us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.033310 (+ 102us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.033383 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":56,"threads_started":1}
I20250903 14:27:00.034053 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.030095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41991 (local address 127.0.0.1:60706)
0903 14:27:00.030447 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.030459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.030528 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.031443 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.031446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.031497 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:00.031682 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.031688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032823 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.032830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.033679 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.033689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.033793 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.033827 (+ 34us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.033882 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.033935 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"spinlock_wait_cycles":63104,"thread_start_us":88,"threads_started":1}
I20250903 14:27:00.034202 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.030482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43373 (local address 127.0.0.1:36340)
0903 14:27:00.030710 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.030723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.030790 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.031442 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.031444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.031481 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:00.031667 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.031672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032966 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.032970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.033819 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.033829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.033938 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.033960 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.034013 (+ 53us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.034070 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"spinlock_wait_cycles":1408,"thread_start_us":71,"threads_started":1}
I20250903 14:27:00.034708 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.030764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.19.65.131:41991)
0903 14:27:00.031053 (+ 289us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.031057 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.031072 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:00.031122 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.031126 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.031241 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.031346 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:00.031784 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032717 (+ 933us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.033870 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.034390 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.034471 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.034515 (+ 44us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.034587 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":126,"threads_started":1}
I20250903 14:27:00.035107 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.030108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.19.65.130:40671)
0903 14:27:00.030861 (+ 753us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.030870 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.030891 (+ 21us) server_negotiation.cc:408] Connection header received
0903 14:27:00.030968 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.030972 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.031134 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.031660 (+ 526us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:00.031779 (+ 119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032359 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.033214 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.034756 (+ 1542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.034849 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.034904 (+ 55us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.034989 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
I20250903 14:27:00.035202 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.030631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.19.65.129:43373)
0903 14:27:00.030942 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.030947 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.031071 (+ 124us) server_negotiation.cc:408] Connection header received
0903 14:27:00.031155 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.031161 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.031309 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.031821 (+ 512us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:00.031951 (+ 130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.032878 (+ 927us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.034384 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.034862 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.034939 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.034997 (+ 58us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.035058 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20250903 14:27:00.036587 24797 tablet_service.cc:1468] Processing CreateTablet for tablet d7e5e2ecfc184a249facf50451cc116e (DEFAULT_TABLE table=test-table [id=47f9b79c785a44708a4ce4cf96b269fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.036962 24527 tablet_service.cc:1468] Processing CreateTablet for tablet d7e5e2ecfc184a249facf50451cc116e (DEFAULT_TABLE table=test-table [id=47f9b79c785a44708a4ce4cf96b269fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.036962 24662 tablet_service.cc:1468] Processing CreateTablet for tablet d7e5e2ecfc184a249facf50451cc116e (DEFAULT_TABLE table=test-table [id=47f9b79c785a44708a4ce4cf96b269fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.037315 24797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e5e2ecfc184a249facf50451cc116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.037771 24662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e5e2ecfc184a249facf50451cc116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.037771 24527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e5e2ecfc184a249facf50451cc116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.042004 24880 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Bootstrap starting.
I20250903 14:27:00.043294 24881 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Bootstrap starting.
I20250903 14:27:00.044273 24880 tablet_bootstrap.cc:654] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.045017 24880 log.cc:826] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.045171 24881 tablet_bootstrap.cc:654] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.045624 24882 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Bootstrap starting.
I20250903 14:27:00.045874 24881 log.cc:826] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.046509 24880 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: No bootstrap required, opened a new log
I20250903 14:27:00.046671 24880 ts_tablet_manager.cc:1397] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:27:00.047210 24882 tablet_bootstrap.cc:654] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.047384 24881 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: No bootstrap required, opened a new log
I20250903 14:27:00.047535 24881 ts_tablet_manager.cc:1397] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:00.047784 24882 log.cc:826] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.048938 24882 tablet_bootstrap.cc:492] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: No bootstrap required, opened a new log
I20250903 14:27:00.049084 24882 ts_tablet_manager.cc:1397] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:00.049757 24880 raft_consensus.cc:357] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.049943 24880 raft_consensus.cc:383] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.049996 24880 raft_consensus.cc:738] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab87062cc0344496a09336cd4a112afd, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.050377 24880 consensus_queue.cc:260] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.050546 24881 raft_consensus.cc:357] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.050733 24881 raft_consensus.cc:383] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.050791 24881 raft_consensus.cc:738] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f06a16a69f247d19c91ff93e32bd714, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.051136 24863 heartbeater.cc:499] Master 127.19.65.190:38301 was elected leader, sending a full tablet report...
I20250903 14:27:00.051234 24880 ts_tablet_manager.cc:1428] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 14:27:00.051754 24881 consensus_queue.cc:260] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.051819 24882 raft_consensus.cc:357] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.052007 24882 raft_consensus.cc:383] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.052064 24882 raft_consensus.cc:738] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293b1f0bee3e4938b5bdf1730c087483, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.052407 24882 consensus_queue.cc:260] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.052474 24881 ts_tablet_manager.cc:1428] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250903 14:27:00.052755 24728 heartbeater.cc:499] Master 127.19.65.190:38301 was elected leader, sending a full tablet report...
I20250903 14:27:00.053385 24882 ts_tablet_manager.cc:1428] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250903 14:27:00.053755 24593 heartbeater.cc:499] Master 127.19.65.190:38301 was elected leader, sending a full tablet report...
W20250903 14:27:00.061116 24729 tablet.cc:2378] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 14:27:00.122385 24594 tablet.cc:2378] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:00.152766 24888 raft_consensus.cc:491] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:00.152940 24888 raft_consensus.cc:513] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.153990 24888 leader_election.cc:290] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f06a16a69f247d19c91ff93e32bd714 (127.19.65.130:40671), ab87062cc0344496a09336cd4a112afd (127.19.65.131:41991)
I20250903 14:27:00.157292 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.154193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41991 (local address 127.19.65.129:43183)
0903 14:27:00.154579 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.154592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.154689 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.154898 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.154901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.154935 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:00.155207 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.155214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.156079 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.156082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.156915 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.156923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.157027 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.157045 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.157094 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.157148 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":32,"thread_start_us":75,"threads_started":1}
I20250903 14:27:00.157297 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.154193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:40671 (local address 127.19.65.129:33211)
0903 14:27:00.154526 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.154540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.154636 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.154878 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.154882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.154901 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:00.155207 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.155214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.156081 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.156083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:00.156915 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:00.156923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.157026 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.157045 (+ 19us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.157100 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.157148 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":92,"threads_started":1}
I20250903 14:27:00.157701 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.154287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33211 (local address 127.19.65.130:40671)
0903 14:27:00.154506 (+ 219us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.154511 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.154608 (+ 97us) server_negotiation.cc:408] Connection header received
0903 14:27:00.154728 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.154733 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.154784 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.154865 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:00.155315 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.155986 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.157023 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.157456 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.157510 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.157553 (+ 43us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.157601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20250903 14:27:00.157701 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.154286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:43183 (local address 127.19.65.131:41991)
0903 14:27:00.154428 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.154435 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.154655 (+ 220us) server_negotiation.cc:408] Connection header received
0903 14:27:00.154766 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.154769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.154822 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.154888 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:00.155315 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.155981 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.157023 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.157456 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.157510 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.157553 (+ 43us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.157601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 14:27:00.158181 24817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e5e2ecfc184a249facf50451cc116e" candidate_uuid: "293b1f0bee3e4938b5bdf1730c087483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87062cc0344496a09336cd4a112afd" is_pre_election: true
I20250903 14:27:00.158180 24682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e5e2ecfc184a249facf50451cc116e" candidate_uuid: "293b1f0bee3e4938b5bdf1730c087483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f06a16a69f247d19c91ff93e32bd714" is_pre_election: true
I20250903 14:27:00.158401 24682 raft_consensus.cc:2466] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293b1f0bee3e4938b5bdf1730c087483 in term 0.
I20250903 14:27:00.158401 24817 raft_consensus.cc:2466] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293b1f0bee3e4938b5bdf1730c087483 in term 0.
I20250903 14:27:00.158802 24483 leader_election.cc:304] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293b1f0bee3e4938b5bdf1730c087483, ab87062cc0344496a09336cd4a112afd; no voters:
I20250903 14:27:00.159003 24888 raft_consensus.cc:2802] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:00.159071 24888 raft_consensus.cc:491] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:00.159098 24888 raft_consensus.cc:3058] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:00.159850 24888 raft_consensus.cc:513] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.160113 24888 leader_election.cc:290] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [CANDIDATE]: Term 1 election: Requested vote from peers 9f06a16a69f247d19c91ff93e32bd714 (127.19.65.130:40671), ab87062cc0344496a09336cd4a112afd (127.19.65.131:41991)
I20250903 14:27:00.160413 24682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e5e2ecfc184a249facf50451cc116e" candidate_uuid: "293b1f0bee3e4938b5bdf1730c087483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f06a16a69f247d19c91ff93e32bd714"
I20250903 14:27:00.160413 24817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e5e2ecfc184a249facf50451cc116e" candidate_uuid: "293b1f0bee3e4938b5bdf1730c087483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87062cc0344496a09336cd4a112afd"
I20250903 14:27:00.160528 24817 raft_consensus.cc:3058] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:00.160528 24682 raft_consensus.cc:3058] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:00.161609 24682 raft_consensus.cc:2466] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293b1f0bee3e4938b5bdf1730c087483 in term 1.
I20250903 14:27:00.161608 24817 raft_consensus.cc:2466] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293b1f0bee3e4938b5bdf1730c087483 in term 1.
I20250903 14:27:00.161887 24481 leader_election.cc:304] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293b1f0bee3e4938b5bdf1730c087483, 9f06a16a69f247d19c91ff93e32bd714; no voters:
I20250903 14:27:00.162057 24888 raft_consensus.cc:2802] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:00.162258 24888 raft_consensus.cc:695] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [term 1 LEADER]: Becoming Leader. State: Replica: 293b1f0bee3e4938b5bdf1730c087483, State: Running, Role: LEADER
I20250903 14:27:00.162505 24888 consensus_queue.cc:237] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } }
I20250903 14:27:00.165218 24401 catalog_manager.cc:5582] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 293b1f0bee3e4938b5bdf1730c087483 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "293b1f0bee3e4938b5bdf1730c087483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293b1f0bee3e4938b5bdf1730c087483" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43373 } health_report { overall_health: HEALTHY } } }
I20250903 14:27:00.180859 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.178713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.19.65.129:43373)
0903 14:27:00.178802 (+ 89us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.178805 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.178816 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:00.178930 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.178932 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.178972 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.179040 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:00.179303 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.179682 (+ 379us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:00.180160 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:00.180263 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:00.180326 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:00.180651 (+ 325us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:00.180729 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.180756 (+ 27us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.180788 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250903 14:27:00.185187 24817 raft_consensus.cc:1273] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 1 FOLLOWER]: Refusing update from remote peer 293b1f0bee3e4938b5bdf1730c087483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:00.185187 24682 raft_consensus.cc:1273] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 1 FOLLOWER]: Refusing update from remote peer 293b1f0bee3e4938b5bdf1730c087483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:00.185742 24891 consensus_queue.cc:1035] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab87062cc0344496a09336cd4a112afd" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:00.185887 24888 consensus_queue.cc:1035] T d7e5e2ecfc184a249facf50451cc116e P 293b1f0bee3e4938b5bdf1730c087483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f06a16a69f247d19c91ff93e32bd714" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:00.190929 24894 mvcc.cc:204] Tried to move back new op lower bound from 7196301804273020928 to 7196301804187807744. Current Snapshot: MvccSnapshot[applied={T|T < 7196301804273020928}]
I20250903 14:27:00.192595 24895 mvcc.cc:204] Tried to move back new op lower bound from 7196301804273020928 to 7196301804187807744. Current Snapshot: MvccSnapshot[applied={T|T < 7196301804273020928}]
I20250903 14:27:00.198412 24400 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:00.198524 24400 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:00.200331 24400 catalog_manager.cc:5869] T 00000000000000000000000000000000 P da58b4a599a0485e852d302eadf54c22: Sending DeleteTablet for 3 replicas of tablet d7e5e2ecfc184a249facf50451cc116e
I20250903 14:27:00.200922 24662 tablet_service.cc:1515] Processing DeleteTablet for tablet d7e5e2ecfc184a249facf50451cc116e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:41418
I20250903 14:27:00.200973 24527 tablet_service.cc:1515] Processing DeleteTablet for tablet d7e5e2ecfc184a249facf50451cc116e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:36340
I20250903 14:27:00.200973 24797 tablet_service.cc:1515] Processing DeleteTablet for tablet d7e5e2ecfc184a249facf50451cc116e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:60706
I20250903 14:27:00.201422 24905 tablet_replica.cc:331] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: stopping tablet replica
I20250903 14:27:00.201584 24903 tablet_replica.cc:331] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: stopping tablet replica
I20250903 14:27:00.201749 24905 raft_consensus.cc:2241] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:00.201748 24903 raft_consensus.cc:2241] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:00.201808 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24465
I20250903 14:27:00.203147 24903 raft_consensus.cc:2270] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:00.204178 24903 ts_tablet_manager.cc:1905] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:00.206022 24903 ts_tablet_manager.cc:1918] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:00.206105 24903 log.cc:1199] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/d7e5e2ecfc184a249facf50451cc116e
I20250903 14:27:00.206097 24905 raft_consensus.cc:2270] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:00.206410 24903 ts_tablet_manager.cc:1939] T d7e5e2ecfc184a249facf50451cc116e P ab87062cc0344496a09336cd4a112afd: Deleting consensus metadata
I20250903 14:27:00.207248 24388 catalog_manager.cc:4928] TS ab87062cc0344496a09336cd4a112afd (127.19.65.131:41991): tablet d7e5e2ecfc184a249facf50451cc116e (table test-table [id=47f9b79c785a44708a4ce4cf96b269fe]) successfully deleted
I20250903 14:27:00.207309 24905 ts_tablet_manager.cc:1905] T d7e5e2ecfc184a249facf50451cc116e P 9f06a16a69f247d19c91ff93e32bd714: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 14:27:00.207836 24388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:43373 (error 108)
I20250903 14:27:00.208508 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24600
I20250903 14:27:00.208632 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.208397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43373 (local address 127.0.0.1:36352)
0903 14:27:00.208507 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.208552 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20250903 14:27:00.210242 24388 catalog_manager.cc:4671] TS 293b1f0bee3e4938b5bdf1730c087483 (127.19.65.129:43373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7e5e2ecfc184a249facf50451cc116e: Network error: Client connection negotiation failed: client connection to 127.19.65.129:43373: connect: Connection refused (error 111)
I20250903 14:27:00.213521 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24735
I20250903 14:27:00.213948 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.213658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:40671 (local address 127.0.0.1:41428)
0903 14:27:00.213831 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.213876 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.130:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20250903 14:27:00.218492 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24370
2025-09-03T14:27:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2845 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-09-03T14:27:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:00Z Disabled control of system clock
I20250903 14:27:00.244606 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36131
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 14:27:00.332708 24910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:00.332867 24910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:00.332909 24910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:00.335758 24910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:00.335803 24910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:00.335822 24910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:00.335839 24910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:00.335852 24910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:00.339004 24910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:00.339735 24910 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:00.340575 24910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:00.344882 24918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.344933 24916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.345156 24910 server_base.cc:1047] running on GCE node
W20250903 14:27:00.344878 24915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.345630 24910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:00.346288 24910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:00.347471 24910 hybrid_clock.cc:648] HybridClock initialized: now 1756909620347449 us; error 40 us; skew 500 ppm
I20250903 14:27:00.348989 24910 webserver.cc:480] Webserver started at http://127.19.65.190:40005/ using document root <none> and password file <none>
I20250903 14:27:00.349443 24910 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:00.349519 24910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:00.349684 24910 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:00.351018 24910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "ff0557eac55a401bbc27e782917fb0d4"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "f9e85df0944ff9f2068e90ddbe34f322"
server_key_iv: "6b82db78dac180291b594dda607f305d"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.351387 24910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "ff0557eac55a401bbc27e782917fb0d4"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "f9e85df0944ff9f2068e90ddbe34f322"
server_key_iv: "6b82db78dac180291b594dda607f305d"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.354166 24910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250903 14:27:00.355865 24924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.356626 24910 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:27:00.356706 24910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "ff0557eac55a401bbc27e782917fb0d4"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "f9e85df0944ff9f2068e90ddbe34f322"
server_key_iv: "6b82db78dac180291b594dda607f305d"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.356789 24910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:00.360926 24910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:00.361464 24910 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:00.361634 24910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:00.367540 24976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:36131 every 8 connection(s)
I20250903 14:27:00.367544 24910 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36131
I20250903 14:27:00.368343 24910 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:00.370381 24977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.374724 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: Bootstrap starting.
I20250903 14:27:00.376507 24977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.377106 24977 log.cc:826] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.378177 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 24910
I20250903 14:27:00.378283 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:00.378499 19718 external_mini_cluster.cc:1468] Setting key d3c277dabe65d3d82ca4baf7941ed908
I20250903 14:27:00.378789 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: No bootstrap required, opened a new log
I20250903 14:27:00.381052 24977 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } }
I20250903 14:27:00.381197 24977 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.381246 24977 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0557eac55a401bbc27e782917fb0d4, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.381670 24977 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } }
I20250903 14:27:00.381827 24977 raft_consensus.cc:397] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:00.381891 24977 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:00.381999 24977 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:00.382735 24977 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } }
I20250903 14:27:00.382905 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.379407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.19.65.190:36131)
0903 14:27:00.379892 (+ 485us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.379899 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.379923 (+ 24us) server_negotiation.cc:408] Connection header received
0903 14:27:00.380749 (+ 826us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.380761 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.380959 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.381202 (+ 243us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.381925 (+ 723us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.381932 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.381945 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.381957 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.382019 (+ 62us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.382029 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.382124 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.382269 (+ 145us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.382440 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":110,"threads_started":1}
I20250903 14:27:00.383038 24977 leader_election.cc:304] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0557eac55a401bbc27e782917fb0d4; no voters:
I20250903 14:27:00.383263 24977 leader_election.cc:290] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:00.383354 24982 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:00.383533 24982 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [term 1 LEADER]: Becoming Leader. State: Replica: ff0557eac55a401bbc27e782917fb0d4, State: Running, Role: LEADER
I20250903 14:27:00.383769 24982 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } }
I20250903 14:27:00.384142 24977 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:00.384997 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff0557eac55a401bbc27e782917fb0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } } }
I20250903 14:27:00.385102 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:00.385315 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0557eac55a401bbc27e782917fb0d4. Latest consensus state: current_term: 1 leader_uuid: "ff0557eac55a401bbc27e782917fb0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0557eac55a401bbc27e782917fb0d4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36131 } } }
I20250903 14:27:00.385390 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:00.385625 24991 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:00.387794 24991 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:00.392035 24991 catalog_manager.cc:1349] Generated new cluster ID: 30f20e24cc81414b93ed5d10136d2f6c
I20250903 14:27:00.392098 24991 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:00.410274 24991 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:00.410528 24991 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:00.415313 24991 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: Generated new TSK 0
I20250903 14:27:00.415794 24991 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:27:00.418493 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--builtin_ntp_servers=127.19.65.148:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 14:27:00.505940 25001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:00.506093 25001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:00.506132 25001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:00.508874 25001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:00.508919 25001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:00.508996 25001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:00.512077 25001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:00.512813 25001 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:00.513728 25001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:00.519160 25007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.519171 25006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.519217 25009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.519732 25001 server_base.cc:1047] running on GCE node
I20250903 14:27:00.520030 25001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:00.520455 25001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:00.521596 25001 hybrid_clock.cc:648] HybridClock initialized: now 1756909620521588 us; error 21 us; skew 500 ppm
I20250903 14:27:00.523173 25001 webserver.cc:480] Webserver started at http://127.19.65.129:32889/ using document root <none> and password file <none>
I20250903 14:27:00.523660 25001 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:00.523713 25001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:00.523880 25001 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:00.525283 25001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "2b88817e91fd44169e798b04db7db843"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "275d4b702fa3d7220fdc3c7cfb721301"
server_key_iv: "f80f41f8bb71d4cd9e7e09e6a0e4ed17"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.525669 25001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "2b88817e91fd44169e798b04db7db843"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "275d4b702fa3d7220fdc3c7cfb721301"
server_key_iv: "f80f41f8bb71d4cd9e7e09e6a0e4ed17"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.528393 25001 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250903 14:27:00.530128 25015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.530865 25001 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:00.530951 25001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "2b88817e91fd44169e798b04db7db843"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "275d4b702fa3d7220fdc3c7cfb721301"
server_key_iv: "f80f41f8bb71d4cd9e7e09e6a0e4ed17"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.531018 25001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:00.534927 25001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:00.535354 25001 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:00.535478 25001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:00.535856 25001 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:00.536552 25001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:00.536599 25001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.536644 25001 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:00.536671 25001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.544404 25001 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40587
I20250903 14:27:00.544438 25128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:40587 every 8 connection(s)
I20250903 14:27:00.545149 25001 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:00.549254 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.546530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36893 (local address 127.19.65.190:36131)
0903 14:27:00.546637 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.546641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.547360 (+ 719us) server_negotiation.cc:408] Connection header received
0903 14:27:00.547923 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.547925 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.547957 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.548019 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.548878 (+ 859us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.548881 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.548883 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.548893 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.548914 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.548921 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.548992 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.549101 (+ 109us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.549132 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:00.549844 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.546737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36131 (local address 127.19.65.129:36893)
0903 14:27:00.547247 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.547275 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.547783 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.548130 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.548136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.548200 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.548690 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.548692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.548715 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.548760 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.548984 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.548988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.548991 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.549122 (+ 131us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.549251 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
I20250903 14:27:00.550890 25129 heartbeater.cc:344] Connected to a master server at 127.19.65.190:36131
I20250903 14:27:00.551066 25129 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:00.551358 25129 heartbeater.cc:507] Master 127.19.65.190:36131 requested a full tablet report, sending...
I20250903 14:27:00.552508 24941 ts_manager.cc:194] Registered new tserver with Master: 2b88817e91fd44169e798b04db7db843 (127.19.65.129:40587)
I20250903 14:27:00.552651 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25001
I20250903 14:27:00.552752 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:00.552985 19718 external_mini_cluster.cc:1468] Setting key 0d77615a0589fd0825f61656d158392b
I20250903 14:27:00.554666 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--builtin_ntp_servers=127.19.65.148:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 14:27:00.641978 25132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:00.642144 25132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:00.642216 25132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:00.645115 25132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:00.645155 25132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:00.645277 25132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:00.648311 25132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:00.649127 25132 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:00.650027 25132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:00.655275 25140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.655315 25138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.655326 25137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.655464 25132 server_base.cc:1047] running on GCE node
I20250903 14:27:00.655741 25132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:00.656152 25132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:00.657287 25132 hybrid_clock.cc:648] HybridClock initialized: now 1756909620657277 us; error 26 us; skew 500 ppm
I20250903 14:27:00.658682 25132 webserver.cc:480] Webserver started at http://127.19.65.130:40617/ using document root <none> and password file <none>
I20250903 14:27:00.659127 25132 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:00.659180 25132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:00.659327 25132 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:00.660689 25132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "fb02a1e092134b74906119e8dd682f2b"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "66f7897aa0fc727f5bdc5af4ece0bfa5"
server_key_iv: "8f739bf0d2d03942343afc8f7be65280"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.661080 25132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "fb02a1e092134b74906119e8dd682f2b"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "66f7897aa0fc727f5bdc5af4ece0bfa5"
server_key_iv: "8f739bf0d2d03942343afc8f7be65280"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.663724 25132 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:27:00.665448 25146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.666224 25132 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:27:00.666311 25132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "fb02a1e092134b74906119e8dd682f2b"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "66f7897aa0fc727f5bdc5af4ece0bfa5"
server_key_iv: "8f739bf0d2d03942343afc8f7be65280"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.666380 25132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:00.670071 25132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:00.670512 25132 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:00.670656 25132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:00.671041 25132 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:00.671705 25132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:00.671747 25132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.671801 25132 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:00.671825 25132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.679163 25132 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:33933
I20250903 14:27:00.679193 25259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:33933 every 8 connection(s)
I20250903 14:27:00.679839 25132 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:00.683898 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.681093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41413 (local address 127.19.65.190:36131)
0903 14:27:00.681235 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.681239 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.681959 (+ 720us) server_negotiation.cc:408] Connection header received
0903 14:27:00.682558 (+ 599us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.682561 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.682602 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.682670 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.683547 (+ 877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.683550 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.683552 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.683562 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.683582 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.683588 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.683662 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.683763 (+ 101us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.683798 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 14:27:00.684466 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.681298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36131 (local address 127.19.65.130:41413)
0903 14:27:00.681847 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.681875 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.682418 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.682781 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.682787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.682863 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.683354 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.683356 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.683380 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.683426 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.683654 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.683658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.683661 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.683776 (+ 115us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.683919 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20250903 14:27:00.685457 25260 heartbeater.cc:344] Connected to a master server at 127.19.65.190:36131
I20250903 14:27:00.685672 25260 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:00.685954 25260 heartbeater.cc:507] Master 127.19.65.190:36131 requested a full tablet report, sending...
I20250903 14:27:00.686710 24941 ts_manager.cc:194] Registered new tserver with Master: fb02a1e092134b74906119e8dd682f2b (127.19.65.130:33933)
I20250903 14:27:00.688525 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25132
I20250903 14:27:00.688638 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:00.688872 19718 external_mini_cluster.cc:1468] Setting key 4cdda3508ad6585571f670dec6ca958f
I20250903 14:27:00.690421 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--builtin_ntp_servers=127.19.65.148:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 14:27:00.783377 25263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:00.783541 25263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:00.783586 25263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:00.786419 25263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:00.786460 25263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:00.786518 25263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:00.789644 25263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:00.790359 25263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:00.791162 25263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:00.796293 25271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:00.796303 25268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.796427 25263 server_base.cc:1047] running on GCE node
W20250903 14:27:00.796499 25269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:00.796824 25263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:00.797324 25263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:00.798451 25263 hybrid_clock.cc:648] HybridClock initialized: now 1756909620798437 us; error 27 us; skew 500 ppm
I20250903 14:27:00.799799 25263 webserver.cc:480] Webserver started at http://127.19.65.131:44721/ using document root <none> and password file <none>
I20250903 14:27:00.800222 25263 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:00.800267 25263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:00.800420 25263 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:00.801756 25263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "72a558c934444524a3aeed8a50ca12bb"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "d367664e19327c03a21c7d32f9b8e550"
server_key_iv: "d4b2739887a6eeebc8b6f8c05cc8ee0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.802137 25263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "72a558c934444524a3aeed8a50ca12bb"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "d367664e19327c03a21c7d32f9b8e550"
server_key_iv: "d4b2739887a6eeebc8b6f8c05cc8ee0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.804750 25263 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:27:00.806407 25277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.807144 25263 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:00.807227 25263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "72a558c934444524a3aeed8a50ca12bb"
format_stamp: "Formatted at 2025-09-03 14:27:00 on dist-test-slave-4wpb"
server_key: "d367664e19327c03a21c7d32f9b8e550"
server_key_iv: "d4b2739887a6eeebc8b6f8c05cc8ee0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:00.807296 25263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:00.810911 25263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:00.811331 25263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:00.811465 25263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:00.811872 25263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:00.812582 25263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:00.812626 25263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.812672 25263 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:00.812703 25263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:00.820299 25263 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40679
I20250903 14:27:00.820331 25390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:40679 every 8 connection(s)
I20250903 14:27:00.820983 25263 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:00.825044 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.822331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:60631 (local address 127.19.65.190:36131)
0903 14:27:00.822449 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.822452 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.823121 (+ 669us) server_negotiation.cc:408] Connection header received
0903 14:27:00.823753 (+ 632us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.823755 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.823793 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.823861 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.824748 (+ 887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.824750 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.824752 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.824758 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.824770 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.824775 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.824832 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.824916 (+ 84us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.824943 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:27:00.825243 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25263
I20250903 14:27:00.825346 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:00.825595 19718 external_mini_cluster.cc:1468] Setting key f94d4c643318562988365718d392cf7a
I20250903 14:27:00.825642 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.822518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36131 (local address 127.19.65.131:60631)
0903 14:27:00.823008 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.823035 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.823606 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.823986 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.823992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.824069 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.824589 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.824591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.824613 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.824656 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.824828 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.824832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.824835 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.824939 (+ 104us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.825067 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20250903 14:27:00.826682 25391 heartbeater.cc:344] Connected to a master server at 127.19.65.190:36131
I20250903 14:27:00.826853 25391 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:00.827155 25391 heartbeater.cc:507] Master 127.19.65.190:36131 requested a full tablet report, sending...
I20250903 14:27:00.827940 24940 ts_manager.cc:194] Registered new tserver with Master: 72a558c934444524a3aeed8a50ca12bb (127.19.65.131:40679)
I20250903 14:27:00.838593 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:27:00.841419 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.840089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.19.65.190:36131)
0903 14:27:00.840198 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.840202 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.840328 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:27:00.840430 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.840434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.840459 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.840515 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.841073 (+ 558us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.841076 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.841078 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.841087 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.841107 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.841112 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.841206 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.841285 (+ 79us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.841318 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:00.843544 24940 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:00.845606 24940 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:00.854267 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.851552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33933 (local address 127.0.0.1:58418)
0903 14:27:00.851968 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.852001 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.852088 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.852798 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.852803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.852852 (+ 49us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.853549 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.853554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.853567 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.853592 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.853944 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.853951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.853956 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.854055 (+ 99us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.854112 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"spinlock_wait_cycles":1280,"thread_start_us":77,"threads_started":1}
I20250903 14:27:00.854363 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.851774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:40679 (local address 127.0.0.1:48034)
0903 14:27:00.852037 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.852048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.852113 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.852798 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.852803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.852852 (+ 49us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.853326 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.853330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.853360 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.853420 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.854083 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.854087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.854090 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.854173 (+ 83us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.854209 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20250903 14:27:00.854374 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.852260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.19.65.131:40679)
0903 14:27:00.852498 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.852502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.852517 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:00.852550 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.852553 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.852645 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.853103 (+ 458us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.853898 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.853906 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.853917 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.853930 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.853966 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.853977 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.854089 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.854185 (+ 96us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.854240 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20250903 14:27:00.855015 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.852099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.19.65.130:33933)
0903 14:27:00.852415 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.852421 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.852444 (+ 23us) server_negotiation.cc:408] Connection header received
0903 14:27:00.852515 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.852520 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.852645 (+ 125us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.852978 (+ 333us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.853723 (+ 745us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.853729 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.853738 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.853749 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.853817 (+ 68us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.853833 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.854746 (+ 913us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.854790 (+ 44us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.854844 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":142,"threads_started":1}
I20250903 14:27:00.855279 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.851428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40587 (local address 127.0.0.1:41492)
0903 14:27:00.851968 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:00.852001 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:00.852088 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:00.852936 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:00.852938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:00.852951 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:00.853955 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:00.853957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:00.853968 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:00.853989 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:00.855042 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:00.855048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:00.855050 (+ 2us) client_negotiation.cc:769] Sending connection context
0903 14:27:00.855129 (+ 79us) client_negotiation.cc:240] Negotiation successful
0903 14:27:00.855161 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"spinlock_wait_cycles":3456,"thread_start_us":76,"threads_started":1}
I20250903 14:27:00.855711 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:00.851555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.19.65.129:40587)
0903 14:27:00.852555 (+ 1000us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:00.852561 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:00.852582 (+ 21us) server_negotiation.cc:408] Connection header received
0903 14:27:00.852635 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:00.852642 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:00.852778 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:00.853642 (+ 864us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:00.854849 (+ 1207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:00.854853 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:00.854860 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:00.854868 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:00.854904 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:00.854924 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:00.855044 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:00.855507 (+ 463us) server_negotiation.cc:300] Negotiation successful
0903 14:27:00.855566 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":87,"threads_started":1}
I20250903 14:27:00.856709 25325 tablet_service.cc:1468] Processing CreateTablet for tablet 2f36e5910c5747ce9655c10f2de156ec (DEFAULT_TABLE table=test-table [id=fc20a81ce4f44cb9961359bb8f2f45d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.857044 25194 tablet_service.cc:1468] Processing CreateTablet for tablet 2f36e5910c5747ce9655c10f2de156ec (DEFAULT_TABLE table=test-table [id=fc20a81ce4f44cb9961359bb8f2f45d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.857381 25325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36e5910c5747ce9655c10f2de156ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.857841 25194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36e5910c5747ce9655c10f2de156ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.857607 25063 tablet_service.cc:1468] Processing CreateTablet for tablet 2f36e5910c5747ce9655c10f2de156ec (DEFAULT_TABLE table=test-table [id=fc20a81ce4f44cb9961359bb8f2f45d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:00.858318 25063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36e5910c5747ce9655c10f2de156ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:00.861730 25405 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Bootstrap starting.
I20250903 14:27:00.863164 25406 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Bootstrap starting.
I20250903 14:27:00.863569 25405 tablet_bootstrap.cc:654] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.864027 25407 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Bootstrap starting.
I20250903 14:27:00.864430 25405 log.cc:826] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.865005 25406 tablet_bootstrap.cc:654] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.865569 25407 tablet_bootstrap.cc:654] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:00.865694 25406 log.cc:826] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.866008 25405 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: No bootstrap required, opened a new log
I20250903 14:27:00.866160 25405 ts_tablet_manager.cc:1397] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250903 14:27:00.866197 25407 log.cc:826] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:00.867163 25406 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: No bootstrap required, opened a new log
I20250903 14:27:00.867311 25406 ts_tablet_manager.cc:1397] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:00.867892 25407 tablet_bootstrap.cc:492] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: No bootstrap required, opened a new log
I20250903 14:27:00.868062 25407 ts_tablet_manager.cc:1397] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250903 14:27:00.868734 25405 raft_consensus.cc:357] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.868927 25405 raft_consensus.cc:383] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.868981 25405 raft_consensus.cc:738] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a558c934444524a3aeed8a50ca12bb, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.869369 25405 consensus_queue.cc:260] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.869446 25406 raft_consensus.cc:357] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.869668 25406 raft_consensus.cc:383] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.869727 25406 raft_consensus.cc:738] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb02a1e092134b74906119e8dd682f2b, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.870023 25391 heartbeater.cc:499] Master 127.19.65.190:36131 was elected leader, sending a full tablet report...
I20250903 14:27:00.870177 25405 ts_tablet_manager.cc:1428] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20250903 14:27:00.870132 25406 consensus_queue.cc:260] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.870618 25260 heartbeater.cc:499] Master 127.19.65.190:36131 was elected leader, sending a full tablet report...
I20250903 14:27:00.870767 25406 ts_tablet_manager.cc:1428] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20250903 14:27:00.870679 25407 raft_consensus.cc:357] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.870826 25407 raft_consensus.cc:383] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:00.870885 25407 raft_consensus.cc:738] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b88817e91fd44169e798b04db7db843, State: Initialized, Role: FOLLOWER
I20250903 14:27:00.871193 25407 consensus_queue.cc:260] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:00.871706 25129 heartbeater.cc:499] Master 127.19.65.190:36131 was elected leader, sending a full tablet report...
I20250903 14:27:00.871773 25407 ts_tablet_manager.cc:1428] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20250903 14:27:00.930490 25261 tablet.cc:2378] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 14:27:01.046093 25130 tablet.cc:2378] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 14:27:01.071677 25392 tablet.cc:2378] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:01.099820 25411 raft_consensus.cc:491] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:01.099965 25411 raft_consensus.cc:513] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:01.101039 25411 leader_election.cc:290] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b88817e91fd44169e798b04db7db843 (127.19.65.129:40587), fb02a1e092134b74906119e8dd682f2b (127.19.65.130:33933)
I20250903 14:27:01.102658 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.101218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40587 (local address 127.19.65.131:48183)
0903 14:27:01.101398 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:01.101412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:01.101505 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:01.101750 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:01.101752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:01.101765 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:01.102218 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:01.102220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:01.102228 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:01.102246 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:01.102446 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:01.102449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:01.102452 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:01.102522 (+ 70us) client_negotiation.cc:240] Negotiation successful
0903 14:27:01.102554 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20250903 14:27:01.102658 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.101297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48183 (local address 127.19.65.129:40587)
0903 14:27:01.101425 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.101428 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.101445 (+ 17us) server_negotiation.cc:408] Connection header received
0903 14:27:01.101602 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.101605 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.101652 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.101727 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:01.102348 (+ 621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:01.102350 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:01.102352 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:01.102361 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:01.102377 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:01.102383 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:01.102451 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:01.102531 (+ 80us) server_negotiation.cc:300] Negotiation successful
0903 14:27:01.102564 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250903 14:27:01.102839 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.101339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:53335 (local address 127.19.65.130:33933)
0903 14:27:01.101455 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.101458 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.101691 (+ 233us) server_negotiation.cc:408] Connection header received
0903 14:27:01.101776 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.101779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.101820 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.101899 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:01.102518 (+ 619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:01.102521 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:01.102524 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:01.102535 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:01.102553 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:01.102558 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:01.102625 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:01.102695 (+ 70us) server_negotiation.cc:300] Negotiation successful
0903 14:27:01.102727 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:27:01.102861 25414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.101255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33933 (local address 127.19.65.131:53335)
0903 14:27:01.101610 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:01.101621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:01.101696 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:01.101928 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:01.101933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:01.101946 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:01.102381 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:01.102383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:01.102395 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:01.102414 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:01.102621 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:01.102623 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:01.102626 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:01.102688 (+ 62us) client_negotiation.cc:240] Negotiation successful
0903 14:27:01.102717 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":102,"threads_started":1}
I20250903 14:27:01.103201 25083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f36e5910c5747ce9655c10f2de156ec" candidate_uuid: "72a558c934444524a3aeed8a50ca12bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b88817e91fd44169e798b04db7db843" is_pre_election: true
I20250903 14:27:01.103314 25214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f36e5910c5747ce9655c10f2de156ec" candidate_uuid: "72a558c934444524a3aeed8a50ca12bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb02a1e092134b74906119e8dd682f2b" is_pre_election: true
I20250903 14:27:01.103482 25083 raft_consensus.cc:2466] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a558c934444524a3aeed8a50ca12bb in term 0.
I20250903 14:27:01.103528 25214 raft_consensus.cc:2466] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72a558c934444524a3aeed8a50ca12bb in term 0.
I20250903 14:27:01.103902 25279 leader_election.cc:304] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b88817e91fd44169e798b04db7db843, 72a558c934444524a3aeed8a50ca12bb; no voters:
I20250903 14:27:01.104116 25411 raft_consensus.cc:2802] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:01.104183 25411 raft_consensus.cc:491] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:01.104214 25411 raft_consensus.cc:3058] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:01.105068 25411 raft_consensus.cc:513] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:01.105330 25411 leader_election.cc:290] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [CANDIDATE]: Term 1 election: Requested vote from peers 2b88817e91fd44169e798b04db7db843 (127.19.65.129:40587), fb02a1e092134b74906119e8dd682f2b (127.19.65.130:33933)
I20250903 14:27:01.105609 25214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f36e5910c5747ce9655c10f2de156ec" candidate_uuid: "72a558c934444524a3aeed8a50ca12bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb02a1e092134b74906119e8dd682f2b"
I20250903 14:27:01.105608 25083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f36e5910c5747ce9655c10f2de156ec" candidate_uuid: "72a558c934444524a3aeed8a50ca12bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b88817e91fd44169e798b04db7db843"
I20250903 14:27:01.105711 25083 raft_consensus.cc:3058] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:01.105712 25214 raft_consensus.cc:3058] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:01.106870 25083 raft_consensus.cc:2466] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a558c934444524a3aeed8a50ca12bb in term 1.
I20250903 14:27:01.107134 25279 leader_election.cc:304] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b88817e91fd44169e798b04db7db843, 72a558c934444524a3aeed8a50ca12bb; no voters:
I20250903 14:27:01.107287 25214 raft_consensus.cc:2466] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a558c934444524a3aeed8a50ca12bb in term 1.
I20250903 14:27:01.107308 25411 raft_consensus.cc:2802] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:01.107467 25411 raft_consensus.cc:695] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 1 LEADER]: Becoming Leader. State: Replica: 72a558c934444524a3aeed8a50ca12bb, State: Running, Role: LEADER
I20250903 14:27:01.107751 25411 consensus_queue.cc:237] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } }
I20250903 14:27:01.110170 24940 catalog_manager.cc:5582] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 72a558c934444524a3aeed8a50ca12bb (127.19.65.131). New cstate: current_term: 1 leader_uuid: "72a558c934444524a3aeed8a50ca12bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72a558c934444524a3aeed8a50ca12bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40679 } health_report { overall_health: HEALTHY } } }
I20250903 14:27:01.149171 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.147973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.19.65.131:40679)
0903 14:27:01.148080 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.148083 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.148094 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:01.148195 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.148197 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.148242 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.148314 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:01.148905 (+ 591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:01.148907 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:01.148909 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:01.148916 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:01.148928 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:01.148933 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:01.148990 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:01.149065 (+ 75us) server_negotiation.cc:300] Negotiation successful
0903 14:27:01.149092 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 14:27:01.153671 25083 raft_consensus.cc:1273] T 2f36e5910c5747ce9655c10f2de156ec P 2b88817e91fd44169e798b04db7db843 [term 1 FOLLOWER]: Refusing update from remote peer 72a558c934444524a3aeed8a50ca12bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:01.153677 25214 raft_consensus.cc:1273] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 1 FOLLOWER]: Refusing update from remote peer 72a558c934444524a3aeed8a50ca12bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:01.154238 25411 consensus_queue.cc:1035] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb02a1e092134b74906119e8dd682f2b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:01.154383 25415 consensus_queue.cc:1035] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b88817e91fd44169e798b04db7db843" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:01.159443 25418 mvcc.cc:204] Tried to move back new op lower bound from 7196301808239804416 to 7196301808059293696. Current Snapshot: MvccSnapshot[applied={T|T < 7196301808239804416}]
I20250903 14:27:01.160432 25420 mvcc.cc:204] Tried to move back new op lower bound from 7196301808239804416 to 7196301808059293696. Current Snapshot: MvccSnapshot[applied={T|T < 7196301808239804416}]
I20250903 14:27:01.161161 25419 mvcc.cc:204] Tried to move back new op lower bound from 7196301808239804416 to 7196301808059293696. Current Snapshot: MvccSnapshot[applied={T|T < 7196301808239804416}]
I20250903 14:27:01.166786 24940 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46710:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:01.166886 24940 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46710:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:01.168638 24940 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ff0557eac55a401bbc27e782917fb0d4: Sending DeleteTablet for 3 replicas of tablet 2f36e5910c5747ce9655c10f2de156ec
I20250903 14:27:01.169301 25325 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f36e5910c5747ce9655c10f2de156ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:01 UTC) from {username='slave'} at 127.0.0.1:48034
I20250903 14:27:01.169318 25194 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f36e5910c5747ce9655c10f2de156ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:01 UTC) from {username='slave'} at 127.0.0.1:58418
I20250903 14:27:01.169333 25063 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f36e5910c5747ce9655c10f2de156ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:01 UTC) from {username='slave'} at 127.0.0.1:41492
I20250903 14:27:01.169690 25429 tablet_replica.cc:331] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: stopping tablet replica
I20250903 14:27:01.169909 25429 raft_consensus.cc:2241] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:01.170153 25428 tablet_replica.cc:331] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: stopping tablet replica
I20250903 14:27:01.170185 25429 raft_consensus.cc:2270] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:01.170214 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25001
I20250903 14:27:01.171404 25428 raft_consensus.cc:2241] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:01.171626 25428 raft_consensus.cc:2270] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:01.171660 25429 ts_tablet_manager.cc:1905] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:01.173651 25428 ts_tablet_manager.cc:1905] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:01.173645 25429 ts_tablet_manager.cc:1918] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:01.173733 25429 log.cc:1199] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/2f36e5910c5747ce9655c10f2de156ec
I20250903 14:27:01.174077 25429 ts_tablet_manager.cc:1939] T 2f36e5910c5747ce9655c10f2de156ec P 72a558c934444524a3aeed8a50ca12bb: Deleting consensus metadata
I20250903 14:27:01.175079 24928 catalog_manager.cc:4928] TS 72a558c934444524a3aeed8a50ca12bb (127.19.65.131:40679): tablet 2f36e5910c5747ce9655c10f2de156ec (table test-table [id=fc20a81ce4f44cb9961359bb8f2f45d3]) successfully deleted
I20250903 14:27:01.175829 25428 ts_tablet_manager.cc:1918] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:01.175915 25428 log.cc:1199] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/2f36e5910c5747ce9655c10f2de156ec
I20250903 14:27:01.176198 25428 ts_tablet_manager.cc:1939] T 2f36e5910c5747ce9655c10f2de156ec P fb02a1e092134b74906119e8dd682f2b: Deleting consensus metadata
I20250903 14:27:01.176961 24927 catalog_manager.cc:4928] TS fb02a1e092134b74906119e8dd682f2b (127.19.65.130:33933): tablet 2f36e5910c5747ce9655c10f2de156ec (table test-table [id=fc20a81ce4f44cb9961359bb8f2f45d3]) successfully deleted
W20250903 14:27:01.178956 24926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40587 (error 108)
I20250903 14:27:01.179188 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25132
I20250903 14:27:01.180243 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.179972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40587 (local address 127.0.0.1:41504)
0903 14:27:01.180115 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:01.180168 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20250903 14:27:01.180374 24926 catalog_manager.cc:4671] TS 2b88817e91fd44169e798b04db7db843 (127.19.65.129:40587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f36e5910c5747ce9655c10f2de156ec: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40587: connect: Connection refused (error 111)
I20250903 14:27:01.184967 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25263
I20250903 14:27:01.190181 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 24910
2025-09-03T14:27:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (971 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20250903 14:27:01.210371 19718 mini_oidc.cc:150] Starting JWKS server
I20250903 14:27:01.211534 19718 webserver.cc:480] Webserver started at https://127.0.0.1:41997/ using document root <none> and password file <none>
I20250903 14:27:01.211663 19718 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 14:27:01.211863 19718 webserver.cc:480] Webserver started at http://127.0.0.1:42661/ using document root <none> and password file <none>
2025-09-03T14:27:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:01Z Disabled control of system clock
I20250903 14:27:01.219086 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:38041
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:38041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41997/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/testchainca.cert with env {}
W20250903 14:27:01.310446 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:01.310600 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:01.310642 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:01.313359 25437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:01.313405 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:01.313421 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:01.313439 25437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:01.313462 25437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 14:27:01.313503 25437 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 14:27:01.313539 25437 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41997/jwks.json
I20250903 14:27:01.317355 25437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:38041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41997/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:01.318097 25437 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:01.318706 25437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:01.323185 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:01.323189 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:01.323204 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:01.323453 25437 server_base.cc:1047] running on GCE node
I20250903 14:27:01.323900 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:01.324625 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:01.325762 25437 hybrid_clock.cc:648] HybridClock initialized: now 1756909621325749 us; error 30 us; skew 500 ppm
I20250903 14:27:01.327203 25437 webserver.cc:480] Webserver started at http://127.19.65.190:42737/ using document root <none> and password file <none>
I20250903 14:27:01.327639 25437 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:01.327685 25437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:01.327849 25437 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:01.329258 25437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8"
format_stamp: "Formatted at 2025-09-03 14:27:01 on dist-test-slave-4wpb"
server_key: "2284a998f533bfe86b522aff120e28ca"
server_key_iv: "971e4a727c641c493e8f8dea9f478a7b"
server_key_version: "encryptionkey@0"
I20250903 14:27:01.329746 25437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8"
format_stamp: "Formatted at 2025-09-03 14:27:01 on dist-test-slave-4wpb"
server_key: "2284a998f533bfe86b522aff120e28ca"
server_key_iv: "971e4a727c641c493e8f8dea9f478a7b"
server_key_version: "encryptionkey@0"
I20250903 14:27:01.332485 25437 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:01.334403 25451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:01.335263 25437 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:01.335347 25437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8"
format_stamp: "Formatted at 2025-09-03 14:27:01 on dist-test-slave-4wpb"
server_key: "2284a998f533bfe86b522aff120e28ca"
server_key_iv: "971e4a727c641c493e8f8dea9f478a7b"
server_key_version: "encryptionkey@0"
I20250903 14:27:01.335426 25437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:01.357645 25437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:01.358083 25437 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:01.358208 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:01.364369 25437 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:38041
I20250903 14:27:01.364377 25503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:38041 every 8 connection(s)
I20250903 14:27:01.365433 25437 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:01.367630 25504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:01.371989 25504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8: Bootstrap starting.
I20250903 14:27:01.372946 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25437
I20250903 14:27:01.373068 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:01.373311 19718 external_mini_cluster.cc:1468] Setting key 08ae83b2df1995c2417800d5382402e0
I20250903 14:27:01.374382 25504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:01.375022 25504 log.cc:826] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:01.376451 25504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8: No bootstrap required, opened a new log
I20250903 14:27:01.378549 25504 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } }
I20250903 14:27:01.378690 25504 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:01.378722 25504 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0437eb2932974f0a8b5cfd3e21ff4ff8, State: Initialized, Role: FOLLOWER
I20250903 14:27:01.379119 25504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } }
I20250903 14:27:01.379261 25504 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:01.379320 25504 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:01.379426 25504 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:01.379762 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.374179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.19.65.190:38041)
0903 14:27:01.374749 (+ 570us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.374758 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.374790 (+ 32us) server_negotiation.cc:408] Connection header received
0903 14:27:01.375273 (+ 483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.375294 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.375602 (+ 308us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.375869 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:01.376303 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.376924 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:01.377469 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.377736 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:01.378268 (+ 532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:01.378284 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:01.378293 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:01.378633 (+ 340us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:01.378668 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:01.378684 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:01.378859 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:01.378983 (+ 124us) server_negotiation.cc:300] Negotiation successful
0903 14:27:01.379180 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20250903 14:27:01.380293 25504 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } }
I20250903 14:27:01.380560 25504 leader_election.cc:304] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0437eb2932974f0a8b5cfd3e21ff4ff8; no voters:
I20250903 14:27:01.380786 25504 leader_election.cc:290] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:01.380857 25509 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:01.381024 25509 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 0437eb2932974f0a8b5cfd3e21ff4ff8, State: Running, Role: LEADER
I20250903 14:27:01.381255 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } }
I20250903 14:27:01.381719 25504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:01.382105 19718 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 14:27:01.383137 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } } }
I20250903 14:27:01.383231 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:01.384021 25518 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:01.384330 25511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0437eb2932974f0a8b5cfd3e21ff4ff8. Latest consensus state: current_term: 1 leader_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437eb2932974f0a8b5cfd3e21ff4ff8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 38041 } } }
I20250903 14:27:01.384428 25511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:01.386463 25518 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:01.390161 25518 catalog_manager.cc:1349] Generated new cluster ID: ecbbe4a0ca3b40188856bd92e400c743
I20250903 14:27:01.390213 25518 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:01.411820 25518 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:01.412520 25518 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:01.420822 25518 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0437eb2932974f0a8b5cfd3e21ff4ff8: Generated new TSK 0
I20250903 14:27:01.421399 25518 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:27:01.449955 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.429893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.19.65.190:38041)
0903 14:27:01.430004 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.430008 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.430137 (+ 129us) server_negotiation.cc:408] Connection header received
0903 14:27:01.430269 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.430272 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.430322 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.430410 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:01.430689 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.431369 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:01.431944 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.432116 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:01.432195 (+ 79us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:01.449596 (+ 17401us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0903 14:27:01.449680 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:01.449789 (+ 109us) server_negotiation.cc:300] Negotiation successful
0903 14:27:01.449833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"thread_start_us":99,"threads_started":1}
I20250903 14:27:01.458887 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:01.455682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.19.65.190:38041)
0903 14:27:01.455801 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:01.455805 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:01.455862 (+ 57us) server_negotiation.cc:408] Connection header received
0903 14:27:01.455947 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:01.455950 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:01.455986 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:01.456061 (+ 75us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:01.456321 (+ 260us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.457029 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:01.457569 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.457738 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:01.457790 (+ 52us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:01.458619 (+ 829us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0903 14:27:01.458796 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46754: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":45}
W20250903 14:27:01.458938 25507 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46754: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20250903 14:27:01.459497 25540 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:01.455512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:38041 (local address 127.0.0.1:46754)
0903 14:27:01.455794 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:01.455805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:01.455882 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:01.456064 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:01.456066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:01.456070 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0903 14:27:01.456240 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:01.456244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.457120 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:01.457122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:01.457436 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:01.457441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:01.457567 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:01.457583 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 14:27:01.458783 (+ 1200us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0903 14:27:01.458882 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:38041: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20250903 14:27:16.465986 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:16.463171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.19.65.190:38041)
0903 14:27:16.463422 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.463425 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.463440 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:16.463554 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.463556 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.463599 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.463671 (+ 72us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:16.463966 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.464488 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.465073 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.465195 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.465229 (+ 34us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:16.465775 (+ 546us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0903 14:27:16.465893 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40916: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
W20250903 14:27:16.466054 25546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40916: JWT verification failed: token expired
W20250903 14:27:16.466058 25545 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:16.463075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:38041 (local address 127.0.0.1:40916)
0903 14:27:16.463367 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:16.463380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:16.463469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:16.463684 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:16.463686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:16.463691 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0903 14:27:16.463860 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.463865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.464580 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.464582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:16.464977 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.464984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.465084 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.465110 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 14:27:16.465863 (+ 753us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0903 14:27:16.465950 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:38041: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
W20250903 14:27:16.467954 25546 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:16.467348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.19.65.190:38041)
0903 14:27:16.467439 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.467442 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.467555 (+ 113us) server_negotiation.cc:408] Connection header received
0903 14:27:16.467646 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.467650 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.467684 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.467742 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:16.467897 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40926: BlockingRecv error: recv got EOF from 127.0.0.1:40926 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20250903 14:27:16.470837 25556 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.19.65.190:38041
W20250903 14:27:16.471258 25546 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:16.470556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.19.65.190:38041)
0903 14:27:16.470641 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.470643 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.470811 (+ 168us) server_negotiation.cc:408] Connection header received
0903 14:27:16.470981 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.470983 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.471015 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.471065 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:16.471188 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40938: BlockingRecv error: recv got EOF from 127.0.0.1:40938 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20250903 14:27:16.471462 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25437
2025-09-03T14:27:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20250903 14:27:16.510447 19718 mini_oidc.cc:150] Starting JWKS server
I20250903 14:27:16.511185 19718 webserver.cc:480] Webserver started at https://127.0.0.1:40965/ using document root <none> and password file <none>
I20250903 14:27:16.511266 19718 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 14:27:16.511448 19718 webserver.cc:480] Webserver started at http://127.0.0.1:39863/ using document root <none> and password file <none>
2025-09-03T14:27:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:16Z Disabled control of system clock
I20250903 14:27:16.517894 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43037
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/testchainca.cert with env {}
W20250903 14:27:16.615253 25562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:16.615418 25562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:16.615479 25562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:16.618459 25562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:16.618510 25562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:16.618541 25562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:16.618578 25562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:16.618613 25562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 14:27:16.618649 25562 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 14:27:16.618683 25562 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40965/jwks.json
I20250903 14:27:16.622641 25562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:16.623479 25562 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:16.624178 25562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:16.629181 25570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:16.629261 25567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:16.629181 25568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:16.629402 25562 server_base.cc:1047] running on GCE node
I20250903 14:27:16.629869 25562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:16.630600 25562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:16.631739 25562 hybrid_clock.cc:648] HybridClock initialized: now 1756909636631718 us; error 35 us; skew 500 ppm
I20250903 14:27:16.633271 25562 webserver.cc:480] Webserver started at http://127.19.65.190:36251/ using document root <none> and password file <none>
I20250903 14:27:16.633810 25562 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:16.633869 25562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:16.634045 25562 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:16.635560 25562 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "0645afb768d243279e132035da501160"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "2ddc1fa2de801d93d1e7af62695f9bad"
server_key_iv: "e90aa6c7a21c0290023084d65174672e"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.635989 25562 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "0645afb768d243279e132035da501160"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "2ddc1fa2de801d93d1e7af62695f9bad"
server_key_iv: "e90aa6c7a21c0290023084d65174672e"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.638934 25562 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 14:27:16.640760 25576 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:16.641691 25562 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:16.641798 25562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "0645afb768d243279e132035da501160"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "2ddc1fa2de801d93d1e7af62695f9bad"
server_key_iv: "e90aa6c7a21c0290023084d65174672e"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.641871 25562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:16.659304 25562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:16.659749 25562 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:16.659886 25562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:16.665642 25562 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43037
I20250903 14:27:16.665656 25628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:43037 every 8 connection(s)
I20250903 14:27:16.666401 25562 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:16.668470 25629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:16.671484 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25562
I20250903 14:27:16.671586 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:16.671799 19718 external_mini_cluster.cc:1468] Setting key 07f63588f4aa37b9fbcd85484375b187
I20250903 14:27:16.673740 25629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160: Bootstrap starting.
I20250903 14:27:16.675843 25629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:16.676549 25629 log.cc:826] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:16.678094 25629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160: No bootstrap required, opened a new log
I20250903 14:27:16.680056 25629 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } }
I20250903 14:27:16.680218 25629 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:16.680286 25629 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0645afb768d243279e132035da501160, State: Initialized, Role: FOLLOWER
I20250903 14:27:16.680651 25629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } }
I20250903 14:27:16.680763 25629 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:16.680825 25629 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:16.680930 25629 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:16.681680 25629 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } }
I20250903 14:27:16.681949 25629 leader_election.cc:304] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0645afb768d243279e132035da501160; no voters:
I20250903 14:27:16.682180 25629 leader_election.cc:290] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:16.682299 25634 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:16.682476 25634 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [term 1 LEADER]: Becoming Leader. State: Replica: 0645afb768d243279e132035da501160, State: Running, Role: LEADER
I20250903 14:27:16.682936 25634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } }
I20250903 14:27:16.683086 25629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:16.684391 25635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0645afb768d243279e132035da501160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } } }
I20250903 14:27:16.684507 25635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:16.684600 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:16.672789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.19.65.190:43037)
0903 14:27:16.673236 (+ 447us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.673243 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.673288 (+ 45us) server_negotiation.cc:408] Connection header received
0903 14:27:16.673733 (+ 445us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.673749 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.674102 (+ 353us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.674336 (+ 234us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:16.674877 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.675534 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.676253 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.676413 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.682845 (+ 6432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:16.682864 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:16.682874 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:16.683315 (+ 441us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:16.683358 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:16.683374 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:16.683484 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:16.683660 (+ 176us) server_negotiation.cc:300] Negotiation successful
0903 14:27:16.683864 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20250903 14:27:16.684906 25643 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:16.685577 25636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0645afb768d243279e132035da501160. Latest consensus state: current_term: 1 leader_uuid: "0645afb768d243279e132035da501160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645afb768d243279e132035da501160" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43037 } } }
I20250903 14:27:16.685654 25636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:16.686990 19718 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 14:27:16.687690 25643 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:16.692046 25643 catalog_manager.cc:1349] Generated new cluster ID: 1bbaad54448f49458496bd5ae48f3e6f
I20250903 14:27:16.692132 25643 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:16.696743 25643 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:16.697355 25643 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:16.704802 25643 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0645afb768d243279e132035da501160: Generated new TSK 0
I20250903 14:27:16.705330 25643 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250903 14:27:16.735574 25659 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250903 14:27:16.735639 25659 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250903 14:27:16.736037 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:16.714527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.19.65.190:43037)
0903 14:27:16.714646 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.714650 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.714807 (+ 157us) server_negotiation.cc:408] Connection header received
0903 14:27:16.714959 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.714961 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.715006 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.715101 (+ 95us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:16.715436 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.715869 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.716484 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.716669 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.716739 (+ 70us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:16.735715 (+ 18976us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 14:27:16.735919 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20250903 14:27:16.736105 25632 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250903 14:27:16.736115 25658 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:16.714452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43037 (local address 127.0.0.1:54660)
0903 14:27:16.714733 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:16.714745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:16.714841 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:16.715121 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:16.715124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:16.715131 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0903 14:27:16.715337 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.715344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.715989 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.715992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:16.716391 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.716398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.716495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.716533 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 14:27:16.735857 (+ 19324us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 14:27:16.735968 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:43037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":122,"threads_started":1}
I20250903 14:27:16.736702 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25562
2025-09-03T14:27:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20250903 14:27:16.775470 19718 mini_oidc.cc:150] Starting JWKS server
I20250903 14:27:16.776185 19718 webserver.cc:480] Webserver started at https://127.0.0.1:41689/ using document root <none> and password file <none>
I20250903 14:27:16.776262 19718 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 14:27:16.776463 19718 webserver.cc:480] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
2025-09-03T14:27:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:16Z Disabled control of system clock
I20250903 14:27:16.783049 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44977
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41689/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250903 14:27:16.874260 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:16.874430 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:16.874473 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:16.877228 25665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:16.877271 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:16.877285 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:16.877300 25665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:16.877314 25665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 14:27:16.877329 25665 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 14:27:16.877341 25665 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41689/jwks.json
I20250903 14:27:16.880317 25665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41689/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:16.881091 25665 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:16.882009 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:16.886340 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:16.886355 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:16.886617 25665 server_base.cc:1047] running on GCE node
W20250903 14:27:16.886648 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:16.887017 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:16.887753 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:16.888890 25665 hybrid_clock.cc:648] HybridClock initialized: now 1756909636888873 us; error 29 us; skew 500 ppm
I20250903 14:27:16.890352 25665 webserver.cc:480] Webserver started at http://127.19.65.190:41391/ using document root <none> and password file <none>
I20250903 14:27:16.890801 25665 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:16.890853 25665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:16.890998 25665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:16.892378 25665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "e342d1c4f74b5c4342f77bd50acde2cc"
server_key_iv: "6f21e8daa2c518e10b0d18013d67380a"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.892764 25665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "e342d1c4f74b5c4342f77bd50acde2cc"
server_key_iv: "6f21e8daa2c518e10b0d18013d67380a"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.895656 25665 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250903 14:27:16.897347 25679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:16.898293 25665 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 14:27:16.898389 25665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2"
format_stamp: "Formatted at 2025-09-03 14:27:16 on dist-test-slave-4wpb"
server_key: "e342d1c4f74b5c4342f77bd50acde2cc"
server_key_iv: "6f21e8daa2c518e10b0d18013d67380a"
server_key_version: "encryptionkey@0"
I20250903 14:27:16.898510 25665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:16.914034 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:16.914532 25665 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:16.914680 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:16.920205 25665 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44977
I20250903 14:27:16.920215 25731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:44977 every 8 connection(s)
I20250903 14:27:16.921037 25665 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:16.923220 25732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:16.927115 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25665
I20250903 14:27:16.927248 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:16.927461 19718 external_mini_cluster.cc:1468] Setting key c968fbeedd61766968dd51ff20e7c8e6
I20250903 14:27:16.927927 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2: Bootstrap starting.
I20250903 14:27:16.929776 25732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:16.930336 25732 log.cc:826] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:16.931635 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2: No bootstrap required, opened a new log
I20250903 14:27:16.933368 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:16.928457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.19.65.190:44977)
0903 14:27:16.928928 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.928935 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.928962 (+ 27us) server_negotiation.cc:408] Connection header received
0903 14:27:16.929378 (+ 416us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.929392 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.929709 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.929942 (+ 233us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:16.930365 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.930989 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.931548 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.931794 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.932184 (+ 390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:16.932199 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:16.932208 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:16.932489 (+ 281us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:16.932522 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:16.932534 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:16.932614 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:16.932744 (+ 130us) server_negotiation.cc:300] Negotiation successful
0903 14:27:16.932907 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":114,"threads_started":1}
I20250903 14:27:16.933791 25732 raft_consensus.cc:357] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } }
I20250903 14:27:16.933970 25732 raft_consensus.cc:383] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:16.934026 25732 raft_consensus.cc:738] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db20fbcdf4ca4ae9a64cb3675ffba3b2, State: Initialized, Role: FOLLOWER
I20250903 14:27:16.934418 25732 consensus_queue.cc:260] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } }
I20250903 14:27:16.934544 25732 raft_consensus.cc:397] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:16.934607 25732 raft_consensus.cc:491] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:16.934729 25732 raft_consensus.cc:3058] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:16.935492 25732 raft_consensus.cc:513] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } }
I20250903 14:27:16.935621 19718 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 14:27:16.935746 25732 leader_election.cc:304] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db20fbcdf4ca4ae9a64cb3675ffba3b2; no voters:
I20250903 14:27:16.935949 25732 leader_election.cc:290] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:16.936049 25737 raft_consensus.cc:2802] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:16.936172 25737 raft_consensus.cc:695] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [term 1 LEADER]: Becoming Leader. State: Replica: db20fbcdf4ca4ae9a64cb3675ffba3b2, State: Running, Role: LEADER
I20250903 14:27:16.936395 25737 consensus_queue.cc:237] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } }
I20250903 14:27:16.936861 25732 sys_catalog.cc:564] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:16.937330 25737 sys_catalog.cc:455] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db20fbcdf4ca4ae9a64cb3675ffba3b2. Latest consensus state: current_term: 1 leader_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } } }
I20250903 14:27:16.937321 25738 sys_catalog.cc:455] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db20fbcdf4ca4ae9a64cb3675ffba3b2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44977 } } }
I20250903 14:27:16.937544 25738 sys_catalog.cc:458] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:16.937543 25737 sys_catalog.cc:458] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:16.938057 25745 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:16.940434 25745 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:16.944567 25745 catalog_manager.cc:1349] Generated new cluster ID: e3af9296718e4ac485849ad906551944
I20250903 14:27:16.944628 25745 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:16.960858 25745 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:16.961525 25745 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:16.970472 25745 catalog_manager.cc:5955] T 00000000000000000000000000000000 P db20fbcdf4ca4ae9a64cb3675ffba3b2: Generated new TSK 0
I20250903 14:27:16.971053 25745 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250903 14:27:17.002293 25762 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250903 14:27:17.002355 25762 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250903 14:27:17.003312 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:16.982689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.19.65.190:44977)
0903 14:27:16.982786 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:16.982790 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:16.982916 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:27:16.983047 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:16.983049 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:16.983086 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:16.983156 (+ 70us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:16.983455 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.983882 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.984510 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.984695 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.984765 (+ 70us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:17.002974 (+ 18209us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 14:27:17.003197 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20250903 14:27:17.003350 25761 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:16.982625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44977 (local address 127.0.0.1:40236)
0903 14:27:16.982852 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:16.982862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:16.982947 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:16.983167 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:16.983170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:16.983177 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0903 14:27:16.983365 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.983370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.984018 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:16.984021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:16.984409 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:16.984416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:16.984523 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:16.984545 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 14:27:17.003104 (+ 18559us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 14:27:17.003208 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:44977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
W20250903 14:27:17.003366 25735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20250903 14:27:17.003777 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25665
2025-09-03T14:27:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (266 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-09-03T14:27:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:17Z Disabled control of system clock
I20250903 14:27:17.048605 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36751
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36751
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/testchainca.cert with env {}
W20250903 14:27:17.137305 25766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:17.137494 25766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:17.137568 25766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:17.140126 25766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:17.140174 25766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:17.140199 25766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:17.140244 25766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:17.140275 25766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 14:27:17.140306 25766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 14:27:17.140338 25766 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20250903 14:27:17.143864 25766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:17.144757 25766 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:17.145460 25766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:17.149684 25771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:17.149690 25772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:17.149724 25766 server_base.cc:1047] running on GCE node
W20250903 14:27:17.149842 25774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:17.150295 25766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:17.151057 25766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:17.152199 25766 hybrid_clock.cc:648] HybridClock initialized: now 1756909637152173 us; error 39 us; skew 500 ppm
I20250903 14:27:17.153610 25766 webserver.cc:480] Webserver started at http://127.19.65.190:33413/ using document root <none> and password file <none>
I20250903 14:27:17.154078 25766 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:17.154126 25766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:17.154263 25766 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:17.155689 25766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "07f71dbbc600424593da9f937a140378"
format_stamp: "Formatted at 2025-09-03 14:27:17 on dist-test-slave-4wpb"
server_key: "4420f74c00d3947944fa9de487d34679"
server_key_iv: "850b721e304ec656b753e36b5cfb0620"
server_key_version: "encryptionkey@0"
I20250903 14:27:17.156064 25766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "07f71dbbc600424593da9f937a140378"
format_stamp: "Formatted at 2025-09-03 14:27:17 on dist-test-slave-4wpb"
server_key: "4420f74c00d3947944fa9de487d34679"
server_key_iv: "850b721e304ec656b753e36b5cfb0620"
server_key_version: "encryptionkey@0"
I20250903 14:27:17.158700 25766 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20250903 14:27:17.160439 25780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:17.161280 25766 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:27:17.161360 25766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "07f71dbbc600424593da9f937a140378"
format_stamp: "Formatted at 2025-09-03 14:27:17 on dist-test-slave-4wpb"
server_key: "4420f74c00d3947944fa9de487d34679"
server_key_iv: "850b721e304ec656b753e36b5cfb0620"
server_key_version: "encryptionkey@0"
I20250903 14:27:17.161420 25766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:17.172181 25766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:17.172634 25766 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:17.172744 25766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:17.178678 25766 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36751
I20250903 14:27:17.178686 25832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:36751 every 8 connection(s)
I20250903 14:27:17.179441 25766 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:17.181602 25833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:17.182132 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25766
I20250903 14:27:17.182230 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:17.182488 19718 external_mini_cluster.cc:1468] Setting key 6e0add662af9be536ed0b7ceadf96c53
I20250903 14:27:17.186687 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378: Bootstrap starting.
I20250903 14:27:17.188606 25833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:17.189294 25833 log.cc:826] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:17.189761 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:17.183434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.19.65.190:36751)
0903 14:27:17.184442 (+ 1008us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:17.184453 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:17.184500 (+ 47us) server_negotiation.cc:408] Connection header received
0903 14:27:17.185143 (+ 643us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:17.185166 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:17.185602 (+ 436us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:17.185872 (+ 270us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:17.186271 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.187223 (+ 952us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:17.187758 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.187967 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:17.188313 (+ 346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:17.188325 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:17.188335 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:17.188716 (+ 381us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:17.188758 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:17.188778 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:17.188870 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:17.188988 (+ 118us) server_negotiation.cc:300] Negotiation successful
0903 14:27:17.189161 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":490,"threads_started":1}
I20250903 14:27:17.190805 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378: No bootstrap required, opened a new log
I20250903 14:27:17.191906 19718 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 14:27:17.193078 25833 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } }
I20250903 14:27:17.193300 25833 raft_consensus.cc:383] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:17.193368 25833 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f71dbbc600424593da9f937a140378, State: Initialized, Role: FOLLOWER
I20250903 14:27:17.193763 25833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } }
I20250903 14:27:17.193867 25833 raft_consensus.cc:397] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:17.193928 25833 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:17.194048 25833 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:17.194751 25833 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } }
I20250903 14:27:17.195029 25833 leader_election.cc:304] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07f71dbbc600424593da9f937a140378; no voters:
I20250903 14:27:17.195262 25833 leader_election.cc:290] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:17.195344 25838 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:17.195514 25838 raft_consensus.cc:695] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [term 1 LEADER]: Becoming Leader. State: Replica: 07f71dbbc600424593da9f937a140378, State: Running, Role: LEADER
I20250903 14:27:17.195770 25838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } }
I20250903 14:27:17.196130 25833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:17.196725 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f71dbbc600424593da9f937a140378. Latest consensus state: current_term: 1 leader_uuid: "07f71dbbc600424593da9f937a140378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } } }
I20250903 14:27:17.196832 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:17.197140 25847 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:17.198068 25840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07f71dbbc600424593da9f937a140378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f71dbbc600424593da9f937a140378" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36751 } } }
I20250903 14:27:17.198163 25840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:17.199246 25847 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:17.203408 25847 catalog_manager.cc:1349] Generated new cluster ID: 4cf14d31daec453c99def20fb6326a1c
I20250903 14:27:17.203456 25847 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:17.212194 25847 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:17.212884 25847 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:17.219362 25847 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 07f71dbbc600424593da9f937a140378: Generated new TSK 0
I20250903 14:27:17.219841 25847 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250903 14:27:17.329917 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:17.231080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36751 (local address 127.0.0.1:52754)
0903 14:27:17.231301 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:17.231312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:17.231385 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:17.231605 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:17.231607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:17.231613 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0903 14:27:17.231777 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:17.231782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.232599 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:17.232601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:17.232934 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:17.232939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.233009 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:17.233024 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 14:27:17.329649 (+ 96625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0903 14:27:17.329785 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:36751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20250903 14:27:17.329923 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:17.231162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.19.65.190:36751)
0903 14:27:17.231256 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:17.231260 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:17.231367 (+ 107us) server_negotiation.cc:408] Connection header received
0903 14:27:17.231474 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:17.231477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:17.231526 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:17.231610 (+ 84us) server_negotiation.cc:227] Negotiated authn=JWT
0903 14:27:17.231857 (+ 247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.232519 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:17.233011 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:17.233172 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:17.233255 (+ 83us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 14:27:17.329414 (+ 96159us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0903 14:27:17.329740 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52754: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20250903 14:27:17.329986 25836 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52754: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20250903 14:27:17.330655 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25766
2025-09-03T14:27:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (307 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (99 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:17 dist-test-slave-4wpb krb5kdc[25874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:17 dist-test-slave-4wpb krb5kdc[25874](info): set up 2 sockets
Sep 03 14:27:17 dist-test-slave-4wpb krb5kdc[25874](info): commencing operation
krb5kdc: starting...
W20250903 14:27:19.289803 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.744s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:19.397549 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32971
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:19.485895 25890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:19.486063 25890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:19.486135 25890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:19.488903 25890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:19.488951 25890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:19.488981 25890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:19.489008 25890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:19.489043 25890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:19.492362 25890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:19.493100 25890 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:19.493804 25890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:19.498258 25898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.498461 25890 server_base.cc:1047] running on GCE node
W20250903 14:27:19.498510 25895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:19.498577 25896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.498939 25890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:19.499651 25890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:19.500790 25890 hybrid_clock.cc:648] HybridClock initialized: now 1756909639500769 us; error 34 us; skew 500 ppm
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:19.502956 25890 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:27:19.503779 25890 webserver.cc:480] Webserver started at http://127.19.65.190:40393/ using document root <none> and password file <none>
I20250903 14:27:19.504226 25890 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:19.504269 25890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:19.504402 25890 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:19.505838 25890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "b12e1ce8d7874bc5895c72029a47a166"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "f796bda2501fd535f7064b2b6526221f"
server_key_iv: "ff07dca6d824596b5a4fa7f4f4d72017"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.506227 25890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "b12e1ce8d7874bc5895c72029a47a166"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "f796bda2501fd535f7064b2b6526221f"
server_key_iv: "ff07dca6d824596b5a4fa7f4f4d72017"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.508774 25890 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:19.510492 25905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.511263 25890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:19.511344 25890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "b12e1ce8d7874bc5895c72029a47a166"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "f796bda2501fd535f7064b2b6526221f"
server_key_iv: "ff07dca6d824596b5a4fa7f4f4d72017"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.511417 25890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:19.521131 25890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:19.523407 25890 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:19.523532 25890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:19.529294 25890 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:32971
I20250903 14:27:19.529302 25957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:32971 every 8 connection(s)
I20250903 14:27:19.530043 25890 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:19.531342 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25890
I20250903 14:27:19.531468 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:19.531680 19718 external_mini_cluster.cc:1468] Setting key ddbc97887a35ff1fdd2c61014f0c0835
I20250903 14:27:19.532223 25958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:19.537647 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166: Bootstrap starting.
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:19.539500 25958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:19.540038 25958 log.cc:826] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:19.541322 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166: No bootstrap required, opened a new log
I20250903 14:27:19.542140 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:19.532630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.19.65.190:32971)
0903 14:27:19.532917 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:19.532924 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:19.532950 (+ 26us) server_negotiation.cc:408] Connection header received
0903 14:27:19.533482 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:19.533496 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:19.533752 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:19.533973 (+ 221us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:19.535073 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.535727 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:19.536459 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.536621 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:19.538777 (+ 2156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:19.538808 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:19.538816 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:19.538842 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:19.540269 (+ 1427us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.540613 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.540618 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.540622 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.540678 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.540873 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.540876 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.540877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.541113 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:19.541278 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:19.541531 (+ 253us) server_negotiation.cc:300] Negotiation successful
0903 14:27:19.541712 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20250903 14:27:19.543298 25958 raft_consensus.cc:357] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } }
I20250903 14:27:19.543437 25958 raft_consensus.cc:383] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:19.543514 25958 raft_consensus.cc:738] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12e1ce8d7874bc5895c72029a47a166, State: Initialized, Role: FOLLOWER
I20250903 14:27:19.543851 25958 consensus_queue.cc:260] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } }
I20250903 14:27:19.543947 25958 raft_consensus.cc:397] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:19.544003 25958 raft_consensus.cc:491] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:19.544085 25958 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:19.544795 25958 raft_consensus.cc:513] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } }
I20250903 14:27:19.545071 25958 leader_election.cc:304] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b12e1ce8d7874bc5895c72029a47a166; no voters:
I20250903 14:27:19.545310 25958 leader_election.cc:290] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:19.545591 25963 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:19.545953 25963 raft_consensus.cc:695] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [term 1 LEADER]: Becoming Leader. State: Replica: b12e1ce8d7874bc5895c72029a47a166, State: Running, Role: LEADER
I20250903 14:27:19.546160 25958 sys_catalog.cc:564] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:19.546203 25963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } }
I20250903 14:27:19.547271 25964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b12e1ce8d7874bc5895c72029a47a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } } }
I20250903 14:27:19.547358 25964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:19.547669 25972 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:19.547633 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b12e1ce8d7874bc5895c72029a47a166. Latest consensus state: current_term: 1 leader_uuid: "b12e1ce8d7874bc5895c72029a47a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12e1ce8d7874bc5895c72029a47a166" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32971 } } }
I20250903 14:27:19.547806 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:19.549965 25972 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:19.554200 25972 catalog_manager.cc:1349] Generated new cluster ID: 119b82baf2964e9b9d7d478e716ca886
I20250903 14:27:19.554260 25972 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:19.562197 25972 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:19.563126 25972 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:19.571254 25972 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b12e1ce8d7874bc5895c72029a47a166: Generated new TSK 0
I20250903 14:27:19.571826 25972 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:19.609354 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32971
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:19.699915 25986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:19.700064 25986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:19.700109 25986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:19.702749 25986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:19.702795 25986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:19.702884 25986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:19.706405 25986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:19.707154 25986 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:19.707814 25986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:19.713050 25991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:19.713074 25992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.713256 25986 server_base.cc:1047] running on GCE node
W20250903 14:27:19.713436 25994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.713733 25986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:19.714210 25986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:19.715353 25986 hybrid_clock.cc:648] HybridClock initialized: now 1756909639715333 us; error 33 us; skew 500 ppm
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:19.717458 25986 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:27:19.718225 25986 webserver.cc:480] Webserver started at http://127.19.65.129:33093/ using document root <none> and password file <none>
I20250903 14:27:19.718626 25986 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:19.718662 25986 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:19.718778 25986 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:19.720050 25986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "0c7a535bbc594eb4bd29fd243f406ec0"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "72aee68d2f5b1d4938bd796f64531239"
server_key_iv: "56b641c5a18acd18e435be413d036e0c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.720394 25986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "0c7a535bbc594eb4bd29fd243f406ec0"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "72aee68d2f5b1d4938bd796f64531239"
server_key_iv: "56b641c5a18acd18e435be413d036e0c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.723031 25986 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20250903 14:27:19.724820 26001 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.725692 25986 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:19.725767 25986 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "0c7a535bbc594eb4bd29fd243f406ec0"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "72aee68d2f5b1d4938bd796f64531239"
server_key_iv: "56b641c5a18acd18e435be413d036e0c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.725840 25986 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:19.737089 25986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:19.739199 25986 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:19.739327 25986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:19.739724 25986 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:19.740327 25986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:19.740367 25986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.740399 25986 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:19.740432 25986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.747680 25986 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:45065
I20250903 14:27:19.747726 26114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:45065 every 8 connection(s)
I20250903 14:27:19.748399 25986 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:19.753139 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 25986
I20250903 14:27:19.753234 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:19.753424 19718 external_mini_cluster.cc:1468] Setting key 5884cca705713763129753454e793813
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:19.758493 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:19.749721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:51395 (local address 127.19.65.190:32971)
0903 14:27:19.749834 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:19.749837 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:19.750337 (+ 500us) server_negotiation.cc:408] Connection header received
0903 14:27:19.750966 (+ 629us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:19.750969 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:19.751014 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:19.751079 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:19.752077 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.752510 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:19.753504 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.753671 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:19.755562 (+ 1891us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:19.755583 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:19.755589 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:19.755619 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:19.756934 (+ 1315us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.757308 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.757314 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.757318 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.757367 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.757624 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.757629 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.757633 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.757796 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:19.757890 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:19.758233 (+ 343us) server_negotiation.cc:300] Negotiation successful
0903 14:27:19.758347 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:27:19.758906 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:19.749906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32971 (local address 127.19.65.129:51395)
0903 14:27:19.750226 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:19.750254 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:19.750818 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:19.751195 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:19.751201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:19.751493 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:19.751967 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:19.751978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.752591 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:19.752593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:19.753388 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:19.753396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.753572 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:19.754026 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:19.754055 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:19.755434 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:19.757021 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:19.757025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:19.757031 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:19.757217 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:19.757450 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:19.757452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:19.757453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:19.757546 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:19.757878 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:19.757881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:19.758031 (+ 150us) client_negotiation.cc:769] Sending connection context
0903 14:27:19.758185 (+ 154us) client_negotiation.cc:240] Negotiation successful
0903 14:27:19.758376 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20250903 14:27:19.759847 26115 heartbeater.cc:344] Connected to a master server at 127.19.65.190:32971
I20250903 14:27:19.760074 26115 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:19.760579 26115 heartbeater.cc:507] Master 127.19.65.190:32971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:27:19.761813 25922 ts_manager.cc:194] Registered new tserver with Master: 0c7a535bbc594eb4bd29fd243f406ec0 (127.19.65.129:45065)
I20250903 14:27:19.762833 25922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:51395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:19.792039 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32971
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:19.878952 26122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:19.879099 26122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:19.879145 26122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:19.881737 26122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:19.881780 26122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:19.881842 26122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:19.885386 26122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:19.886106 26122 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:19.886727 26122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:19.891638 26128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:19.891604 26130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.891714 26122 server_base.cc:1047] running on GCE node
W20250903 14:27:19.891613 26127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:19.892113 26122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:19.892565 26122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:19.893698 26122 hybrid_clock.cc:648] HybridClock initialized: now 1756909639893682 us; error 34 us; skew 500 ppm
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:19.896037 26122 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:27:19.896845 26122 webserver.cc:480] Webserver started at http://127.19.65.130:36719/ using document root <none> and password file <none>
I20250903 14:27:19.897305 26122 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:19.897351 26122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:19.897549 26122 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:19.898959 26122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "ec9151c9b75c47fcbb742c318e6cb52c"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "048b89c138ebda43cbb30ca8695fbaed"
server_key_iv: "18ea62b396e806860c3d19c910d0b57c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.899314 26122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "ec9151c9b75c47fcbb742c318e6cb52c"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "048b89c138ebda43cbb30ca8695fbaed"
server_key_iv: "18ea62b396e806860c3d19c910d0b57c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.902043 26122 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:27:19.903870 26137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.904769 26122 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:19.904865 26122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "ec9151c9b75c47fcbb742c318e6cb52c"
format_stamp: "Formatted at 2025-09-03 14:27:19 on dist-test-slave-4wpb"
server_key: "048b89c138ebda43cbb30ca8695fbaed"
server_key_iv: "18ea62b396e806860c3d19c910d0b57c"
server_key_version: "encryptionkey@0"
I20250903 14:27:19.904942 26122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:19.924540 26122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:19.926689 26122 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:19.926831 26122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:19.927243 26122 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:19.927908 26122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:19.927950 26122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.927996 26122 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:19.928025 26122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:19.935477 26122 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:44927
I20250903 14:27:19.935493 26250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:44927 every 8 connection(s)
I20250903 14:27:19.936149 26122 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
Sep 03 14:27:19 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:19.946041 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26122
I20250903 14:27:19.946075 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:19.937420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50399 (local address 127.19.65.190:32971)
0903 14:27:19.937552 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:19.937555 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:19.938063 (+ 508us) server_negotiation.cc:408] Connection header received
0903 14:27:19.938765 (+ 702us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:19.938769 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:19.938829 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:19.938902 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:19.939922 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.940356 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:19.941264 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.941446 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:19.943192 (+ 1746us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:19.943208 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:19.943210 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:19.943234 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:19.944396 (+ 1162us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.944819 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.944823 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.944825 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.944869 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:19.945114 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:19.945117 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:19.945119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:19.945292 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:19.945388 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:19.945806 (+ 418us) server_negotiation.cc:300] Negotiation successful
0903 14:27:19.945930 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:19.946146 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:19.946388 19718 external_mini_cluster.cc:1468] Setting key 2ea1a3eb12c1f069e1992682437590c7
I20250903 14:27:19.946532 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:19.937612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32971 (local address 127.19.65.130:50399)
0903 14:27:19.937952 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:19.937981 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:19.938590 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:19.939040 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:19.939047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:19.939357 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:19.939793 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:19.939802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.940460 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:19.940463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:19.941161 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:19.941168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:19.941323 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:19.941900 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:19.941919 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:19.943056 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:19.944506 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:19.944511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:19.944521 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:19.944724 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:19.944952 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:19.944957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:19.944959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:19.945030 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:19.945398 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:19.945404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:19.945615 (+ 211us) client_negotiation.cc:769] Sending connection context
0903 14:27:19.945762 (+ 147us) client_negotiation.cc:240] Negotiation successful
0903 14:27:19.946001 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20250903 14:27:19.947341 26251 heartbeater.cc:344] Connected to a master server at 127.19.65.190:32971
I20250903 14:27:19.947558 26251 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:19.948002 26251 heartbeater.cc:507] Master 127.19.65.190:32971 requested a full tablet report, sending...
I20250903 14:27:19.948865 25921 ts_manager.cc:194] Registered new tserver with Master: ec9151c9b75c47fcbb742c318e6cb52c (127.19.65.130:44927)
I20250903 14:27:19.949348 25921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:50399
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:19.985484 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32971
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:20.073236 26258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:20.073385 26258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:20.073455 26258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:20.076280 26258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:20.076324 26258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:20.076386 26258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:20.079684 26258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:20.080435 26258 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:20.081126 26258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:20.086268 26264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:20.086303 26263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:20.086354 26258 server_base.cc:1047] running on GCE node
W20250903 14:27:20.086272 26266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:20.086793 26258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:20.087205 26258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:20.088338 26258 hybrid_clock.cc:648] HybridClock initialized: now 1756909640088327 us; error 25 us; skew 500 ppm
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.090524 26258 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:27:20.091377 26258 webserver.cc:480] Webserver started at http://127.19.65.131:34439/ using document root <none> and password file <none>
I20250903 14:27:20.091774 26258 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:20.091811 26258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:20.091939 26258 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:20.093230 26258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "b1332ced4bc645488f32443c3e0a63d7"
format_stamp: "Formatted at 2025-09-03 14:27:20 on dist-test-slave-4wpb"
server_key: "72b560ca0215bc3ce1e0063671422dbd"
server_key_iv: "d9cd17bef7c704043ab3f61ec3abb7ec"
server_key_version: "encryptionkey@0"
I20250903 14:27:20.093624 26258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "b1332ced4bc645488f32443c3e0a63d7"
format_stamp: "Formatted at 2025-09-03 14:27:20 on dist-test-slave-4wpb"
server_key: "72b560ca0215bc3ce1e0063671422dbd"
server_key_iv: "d9cd17bef7c704043ab3f61ec3abb7ec"
server_key_version: "encryptionkey@0"
I20250903 14:27:20.096383 26258 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 14:27:20.098026 26273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:20.098785 26258 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:20.098874 26258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "b1332ced4bc645488f32443c3e0a63d7"
format_stamp: "Formatted at 2025-09-03 14:27:20 on dist-test-slave-4wpb"
server_key: "72b560ca0215bc3ce1e0063671422dbd"
server_key_iv: "d9cd17bef7c704043ab3f61ec3abb7ec"
server_key_version: "encryptionkey@0"
I20250903 14:27:20.098944 26258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:20.107751 26258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:20.109889 26258 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:20.110028 26258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:20.110456 26258 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:20.111099 26258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:20.111141 26258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:20.111199 26258 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:20.111224 26258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:20.118551 26258 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40403
I20250903 14:27:20.118597 26386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:40403 every 8 connection(s)
I20250903 14:27:20.119287 26258 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:20.128599 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:20.120685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:39847 (local address 127.19.65.190:32971)
0903 14:27:20.120812 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.120817 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.121433 (+ 616us) server_negotiation.cc:408] Connection header received
0903 14:27:20.122089 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.122092 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.122136 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.122200 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.123140 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.123578 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:20.124529 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.124650 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:20.126369 (+ 1719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:20.126384 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:20.126386 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:20.126406 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:20.127424 (+ 1018us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.127773 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.127775 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.127776 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.127805 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.127966 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.127968 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.127969 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.128070 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:20.128128 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:20.128427 (+ 299us) server_negotiation.cc:300] Negotiation successful
0903 14:27:20.128499 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:27:20.129117 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:20.120920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32971 (local address 127.19.65.131:39847)
0903 14:27:20.121308 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:20.121349 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:20.121932 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:20.122317 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:20.122322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:20.122601 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:20.123024 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:20.123038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.123661 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:20.123663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:20.124444 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:20.124451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.124585 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:20.125057 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:20.125078 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:20.126257 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:20.127513 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:20.127517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:20.127524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:20.127703 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:20.127856 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:20.127857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:20.127859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:20.127915 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:20.128127 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:20.128130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:20.128279 (+ 149us) client_negotiation.cc:769] Sending connection context
0903 14:27:20.128405 (+ 126us) client_negotiation.cc:240] Negotiation successful
0903 14:27:20.128564 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20250903 14:27:20.129345 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26258
I20250903 14:27:20.129436 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:20.129686 19718 external_mini_cluster.cc:1468] Setting key 589f4ae0283f9616cbca2c1c5b680797
I20250903 14:27:20.130053 26387 heartbeater.cc:344] Connected to a master server at 127.19.65.190:32971
I20250903 14:27:20.130249 26387 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:20.130697 26387 heartbeater.cc:507] Master 127.19.65.190:32971 requested a full tablet report, sending...
I20250903 14:27:20.131537 25921 ts_manager.cc:194] Registered new tserver with Master: b1332ced4bc645488f32443c3e0a63d7 (127.19.65.131:40403)
I20250903 14:27:20.131984 25921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:39847
I20250903 14:27:20.142800 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/test-admin.keytab.
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.155171 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20250903 14:27:20.157332 26394 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:20.156200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.19.65.129:45065)
0903 14:27:20.156499 (+ 299us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.156503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.156520 (+ 17us) server_negotiation.cc:408] Connection header received
0903 14:27:20.156559 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.156563 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.156678 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.156828 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.157208 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54198: BlockingRecv error: recv got EOF from 127.0.0.1:54198 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":120,"threads_started":1}
W20250903 14:27:20.157438 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc' of type FILE
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.158282 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.158993 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:27:20.166131 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:20.159637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.19.65.129:45065)
0903 14:27:20.159737 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.159740 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.159819 (+ 79us) server_negotiation.cc:408] Connection header received
0903 14:27:20.159888 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.159890 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.159921 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.159978 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.160465 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.161060 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:20.161566 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.161719 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:20.163553 (+ 1834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:20.163572 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:20.163579 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:20.163598 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:20.164996 (+ 1398us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.165351 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.165355 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.165356 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.165393 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.165577 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.165579 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.165580 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.165712 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:20.165828 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:20.165950 (+ 122us) server_negotiation.cc:300] Negotiation successful
0903 14:27:20.166039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20250903 14:27:20.168756 26394 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:20.168001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.19.65.129:45065)
0903 14:27:20.168094 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.168098 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.168154 (+ 56us) server_negotiation.cc:408] Connection header received
0903 14:27:20.168271 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.168273 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.168301 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.168354 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.168689 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54220: BlockingRecv error: recv got EOF from 127.0.0.1:54220 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20250903 14:27:20.168855 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc' of type FILE
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.169652 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.170359 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:27:20.176569 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:20.170992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.19.65.129:45065)
0903 14:27:20.171094 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.171097 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.171166 (+ 69us) server_negotiation.cc:408] Connection header received
0903 14:27:20.171243 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.171245 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.171274 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.171342 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.171748 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.172111 (+ 363us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:20.172542 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.172638 (+ 96us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:20.174295 (+ 1657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:20.174309 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:20.174311 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:20.174331 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:20.175447 (+ 1116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.175750 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.175753 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.175754 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.175786 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.175964 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.175966 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.175968 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.176062 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:20.176155 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:20.176354 (+ 199us) server_negotiation.cc:300] Negotiation successful
0903 14:27:20.176450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20250903 14:27:20.178517 26394 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:20.177851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.19.65.129:45065)
0903 14:27:20.177952 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.177955 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.178027 (+ 72us) server_negotiation.cc:408] Connection header received
0903 14:27:20.178106 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.178108 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.178134 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.178184 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.178448 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54228: BlockingRecv error: recv got EOF from 127.0.0.1:54228 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20250903 14:27:20.178640 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756909585842267-19718-0/krb5kdc/krb5cc' of type FILE
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.179319 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:20.179961 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[25874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20250903 14:27:20.186036 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:20.180518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.19.65.129:45065)
0903 14:27:20.180630 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:20.180634 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:20.180778 (+ 144us) server_negotiation.cc:408] Connection header received
0903 14:27:20.180849 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:20.180851 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:20.180878 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:20.180934 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:20.181403 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.181842 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:20.182339 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:20.182462 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:20.183909 (+ 1447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:20.183920 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:20.183922 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:20.183938 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:20.185089 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.185397 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.185399 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.185400 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.185434 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:20.185618 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:20.185620 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:20.185621 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:20.185706 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:20.185798 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:20.185902 (+ 104us) server_negotiation.cc:300] Negotiation successful
0903 14:27:20.185963 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:27:20.186723 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25986
I20250903 14:27:20.192643 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26122
I20250903 14:27:20.197604 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26258
I20250903 14:27:20.202247 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 25890
2025-09-03T14:27:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2676 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[26407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[26407](info): set up 2 sockets
Sep 03 14:27:20 dist-test-slave-4wpb krb5kdc[26407](info): commencing operation
krb5kdc: starting...
W20250903 14:27:21.969501 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.747s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:22.077028 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37367
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:22.164001 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:22.164144 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:22.164191 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:22.166949 26423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:22.166992 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:22.167011 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:22.167027 26423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:22.167042 26423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:22.170387 26423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:22.171164 26423 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:22.171795 26423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:22.176242 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.176327 26429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.176275 26428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.176506 26423 server_base.cc:1047] running on GCE node
I20250903 14:27:22.176892 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:22.177598 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:22.178726 26423 hybrid_clock.cc:648] HybridClock initialized: now 1756909642178718 us; error 22 us; skew 500 ppm
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:22.180648 26423 init.cc:377] Logged in from keytab as oryx/127.19.65.190@KRBTEST.COM (short username oryx)
I20250903 14:27:22.181489 26423 webserver.cc:480] Webserver started at http://127.19.65.190:32771/ using document root <none> and password file <none>
I20250903 14:27:22.181922 26423 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:22.181972 26423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:22.182117 26423 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:22.183461 26423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "55f880c1f5e3421ab6ec375c6d4932d1"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "4af6e9085a1f476511000ea4987955a7"
server_key_iv: "9e58c3465168e6695afa9f7ed705fcdc"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.183846 26423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "55f880c1f5e3421ab6ec375c6d4932d1"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "4af6e9085a1f476511000ea4987955a7"
server_key_iv: "9e58c3465168e6695afa9f7ed705fcdc"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.186667 26423 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:27:22.188342 26438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.189077 26423 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:27:22.189177 26423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "55f880c1f5e3421ab6ec375c6d4932d1"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "4af6e9085a1f476511000ea4987955a7"
server_key_iv: "9e58c3465168e6695afa9f7ed705fcdc"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.189244 26423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:22.198395 26423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:22.203264 26423 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:22.203398 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:22.209159 26490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:37367 every 8 connection(s)
I20250903 14:27:22.209160 26423 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:37367
I20250903 14:27:22.209941 26423 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:22.210600 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26423
I20250903 14:27:22.210708 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:22.210912 19718 external_mini_cluster.cc:1468] Setting key 60dcc32270356d4f3b2a248eb2537f8d
I20250903 14:27:22.212258 26491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:22.217252 26491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Bootstrap starting.
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:22.219079 26491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:22.219790 26491 log.cc:826] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:22.221144 26491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: No bootstrap required, opened a new log
I20250903 14:27:22.222092 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.211939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.19.65.190:37367)
0903 14:27:22.212313 (+ 374us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:22.212323 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:22.212356 (+ 33us) server_negotiation.cc:408] Connection header received
0903 14:27:22.213607 (+ 1251us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:22.213629 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:22.213939 (+ 310us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:22.214260 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:22.214946 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.215995 (+ 1049us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.216527 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.216753 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.218454 (+ 1701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:22.218489 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:22.218499 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:22.218531 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:22.220188 (+ 1657us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.220528 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.220532 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.220535 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.220590 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.220787 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.220790 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.220791 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.221033 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:22.221180 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:22.221437 (+ 257us) server_negotiation.cc:300] Negotiation successful
0903 14:27:22.221640 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
I20250903 14:27:22.223198 26491 raft_consensus.cc:357] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } }
I20250903 14:27:22.223340 26491 raft_consensus.cc:383] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:22.223407 26491 raft_consensus.cc:738] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f880c1f5e3421ab6ec375c6d4932d1, State: Initialized, Role: FOLLOWER
I20250903 14:27:22.223734 26491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } }
I20250903 14:27:22.223841 26491 raft_consensus.cc:397] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:22.223896 26491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:22.223986 26491 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:22.224685 26491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } }
I20250903 14:27:22.224962 26491 leader_election.cc:304] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f880c1f5e3421ab6ec375c6d4932d1; no voters:
I20250903 14:27:22.225188 26491 leader_election.cc:290] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:22.225279 26496 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:22.225458 26496 raft_consensus.cc:695] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 LEADER]: Becoming Leader. State: Replica: 55f880c1f5e3421ab6ec375c6d4932d1, State: Running, Role: LEADER
I20250903 14:27:22.225737 26496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } }
I20250903 14:27:22.226151 26491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:22.227093 26498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f880c1f5e3421ab6ec375c6d4932d1. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } }
I20250903 14:27:22.227195 26498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:22.227303 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } }
I20250903 14:27:22.227382 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:22.227586 26505 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:22.230086 26505 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:22.234352 26505 catalog_manager.cc:1349] Generated new cluster ID: 8be42f9d5dde470aa2cf5c0895508058
I20250903 14:27:22.234419 26505 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:22.241631 26505 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:22.242291 26505 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:22.251052 26505 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Generated new TSK 0
I20250903 14:27:22.251525 26505 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:22.289132 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37367
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:22.378079 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:22.378288 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:22.378371 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:22.380980 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:22.381032 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:22.381127 26519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:22.384513 26519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:22.385376 26519 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:22.386068 26519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:22.391301 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.391296 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.391448 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.391451 26519 server_base.cc:1047] running on GCE node
I20250903 14:27:22.391779 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:22.392200 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:22.393340 26519 hybrid_clock.cc:648] HybridClock initialized: now 1756909642393319 us; error 34 us; skew 500 ppm
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:22.395620 26519 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20250903 14:27:22.396435 26519 webserver.cc:480] Webserver started at http://127.19.65.129:32997/ using document root <none> and password file <none>
I20250903 14:27:22.396885 26519 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:22.396934 26519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:22.397085 26519 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:22.398557 26519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "5b407477dde04cc58b0049f4251e8073"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "1b3970025db9eb2a5e376e092534198c"
server_key_iv: "0f72c8028efcf43a7b087ed12a984a70"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.398954 26519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "5b407477dde04cc58b0049f4251e8073"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "1b3970025db9eb2a5e376e092534198c"
server_key_iv: "0f72c8028efcf43a7b087ed12a984a70"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.401635 26519 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20250903 14:27:22.403339 26534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.404090 26519 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:27:22.404181 26519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "5b407477dde04cc58b0049f4251e8073"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "1b3970025db9eb2a5e376e092534198c"
server_key_iv: "0f72c8028efcf43a7b087ed12a984a70"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.404271 26519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:22.423470 26519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:22.425981 26519 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:22.426120 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:22.426555 26519 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:22.427218 26519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:22.427258 26519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.427304 26519 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:22.427330 26519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.435585 26519 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:39959
I20250903 14:27:22.435614 26647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:39959 every 8 connection(s)
I20250903 14:27:22.436286 26519 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:22.443135 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26519
I20250903 14:27:22.443229 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:22.443502 19718 external_mini_cluster.cc:1468] Setting key 31135a287793c100741d44230f1e33a6
I20250903 14:27:22.446247 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.437609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:38657 (local address 127.19.65.190:37367)
0903 14:27:22.437724 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:22.437727 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:22.438234 (+ 507us) server_negotiation.cc:408] Connection header received
0903 14:27:22.438899 (+ 665us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:22.438902 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:22.438957 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:22.439021 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:22.440307 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.440742 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.441267 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.441436 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.443025 (+ 1589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:22.443041 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:22.443043 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:22.443064 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:22.444470 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.444992 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.444995 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.444996 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.445036 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.445326 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.445328 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.445329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.445472 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:22.445585 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:22.446055 (+ 470us) server_negotiation.cc:300] Negotiation successful
0903 14:27:22.446139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:27:22.446885 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.437799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:38657)
0903 14:27:22.438125 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:22.438152 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:22.438735 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:22.439158 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:22.439165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:22.439459 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:22.440177 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.440188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.440843 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.440846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:22.441173 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.441179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.441331 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.441791 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:22.441812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:22.442903 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:22.444590 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.444599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.444611 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.444897 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.445126 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.445130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.445132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.445248 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.445598 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:22.445606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:22.445823 (+ 217us) client_negotiation.cc:769] Sending connection context
0903 14:27:22.446039 (+ 216us) client_negotiation.cc:240] Negotiation successful
0903 14:27:22.446294 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20250903 14:27:22.447767 26648 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:22.447962 26648 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:22.448470 26648 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
I20250903 14:27:22.449833 26455 ts_manager.cc:194] Registered new tserver with Master: 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:22.450785 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:38657
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:22.480937 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37367
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:22.569331 26655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:22.569520 26655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:22.569595 26655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:22.572104 26655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:22.572150 26655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:22.572242 26655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:22.575835 26655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:22.576568 26655 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:22.577172 26655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:22.582018 26663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.582089 26660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.582134 26655 server_base.cc:1047] running on GCE node
W20250903 14:27:22.582020 26661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.582507 26655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:22.582913 26655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:22.584050 26655 hybrid_clock.cc:648] HybridClock initialized: now 1756909642584042 us; error 20 us; skew 500 ppm
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:22.586355 26655 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20250903 14:27:22.587265 26655 webserver.cc:480] Webserver started at http://127.19.65.130:39459/ using document root <none> and password file <none>
I20250903 14:27:22.587702 26655 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:22.587752 26655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:22.587904 26655 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:22.589257 26655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "d99be9c94fbc410ebf5f1c2f503e59b0"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "b2c2066b6bf8b8e316162dd23231026a"
server_key_iv: "dc894932b46d9877f2157a61db3d2da7"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.589654 26655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "d99be9c94fbc410ebf5f1c2f503e59b0"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "b2c2066b6bf8b8e316162dd23231026a"
server_key_iv: "dc894932b46d9877f2157a61db3d2da7"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.592176 26655 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:22.593833 26670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.594643 26655 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:27:22.594744 26655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "d99be9c94fbc410ebf5f1c2f503e59b0"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "b2c2066b6bf8b8e316162dd23231026a"
server_key_iv: "dc894932b46d9877f2157a61db3d2da7"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.594820 26655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:22.608873 26655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:22.610802 26655 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:22.610934 26655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:22.611328 26655 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:22.611986 26655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:22.612030 26655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.612076 26655 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:22.612115 26655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.619753 26655 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43183
I20250903 14:27:22.619781 26783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:43183 every 8 connection(s)
I20250903 14:27:22.620429 26655 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:22.624856 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26655
I20250903 14:27:22.624974 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:22.625236 19718 external_mini_cluster.cc:1468] Setting key 98e82c4141d292c93c3c07f8181b2840
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:22.630910 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.621710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33607 (local address 127.19.65.190:37367)
0903 14:27:22.621843 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:22.621847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:22.622370 (+ 523us) server_negotiation.cc:408] Connection header received
0903 14:27:22.623041 (+ 671us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:22.623044 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:22.623098 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:22.623167 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:22.624402 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.625055 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.625763 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.625929 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.627680 (+ 1751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:22.627699 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:22.627706 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:22.627737 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:22.629051 (+ 1314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.629582 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.629587 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.629590 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.629643 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.629968 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.629973 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.629976 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.630156 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:22.630241 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:22.630727 (+ 486us) server_negotiation.cc:300] Negotiation successful
0903 14:27:22.630811 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:27:22.631593 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.621888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:33607)
0903 14:27:22.622256 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:22.622291 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:22.622905 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:22.623271 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:22.623276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:22.623543 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:22.624283 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.624295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.625174 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.625177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:22.625648 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.625657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.625806 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.626230 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:22.626254 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:22.627554 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:22.629152 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.629160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.629171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.629490 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.629737 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.629742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.629745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.629851 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.630258 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:22.630266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:22.630507 (+ 241us) client_negotiation.cc:769] Sending connection context
0903 14:27:22.630706 (+ 199us) client_negotiation.cc:240] Negotiation successful
0903 14:27:22.630935 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20250903 14:27:22.632433 26784 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:22.632608 26784 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:22.632969 26784 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20250903 14:27:22.633793 26455 ts_manager.cc:194] Registered new tserver with Master: d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183)
I20250903 14:27:22.634282 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:33607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:22.662014 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37367
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:22.749413 26791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:22.749576 26791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:22.749622 26791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:22.752214 26791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:22.752259 26791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:22.752321 26791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:22.755813 26791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:22.756533 26791 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:22.757167 26791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:22.762037 26796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.762039 26799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.762239 26791 server_base.cc:1047] running on GCE node
W20250903 14:27:22.762367 26797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.762632 26791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:22.763043 26791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:22.764175 26791 hybrid_clock.cc:648] HybridClock initialized: now 1756909642764164 us; error 26 us; skew 500 ppm
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:22.766471 26791 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20250903 14:27:22.767249 26791 webserver.cc:480] Webserver started at http://127.19.65.131:41003/ using document root <none> and password file <none>
I20250903 14:27:22.767660 26791 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:22.767709 26791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:22.767848 26791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:22.769161 26791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "f8fea8909934484380f011507ceb0a66"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "d3c0f646bb749087ee3ea9ee057b87c8"
server_key_iv: "3940d483f2d4890de3e8334c2499fd9e"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.769577 26791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "f8fea8909934484380f011507ceb0a66"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "d3c0f646bb749087ee3ea9ee057b87c8"
server_key_iv: "3940d483f2d4890de3e8334c2499fd9e"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.772250 26791 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:22.773977 26806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.774842 26791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:22.774938 26791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "f8fea8909934484380f011507ceb0a66"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "d3c0f646bb749087ee3ea9ee057b87c8"
server_key_iv: "3940d483f2d4890de3e8334c2499fd9e"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.775014 26791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:22.784654 26791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:22.786986 26791 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:22.787128 26791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:22.787528 26791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:22.788141 26791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:22.788182 26791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.788225 26791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:22.788254 26791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.795747 26791 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38077
I20250903 14:27:22.795760 26919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:38077 every 8 connection(s)
I20250903 14:27:22.796432 26791 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:22.805963 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26791
I20250903 14:27:22.806052 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:22.806120 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.797730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:49449 (local address 127.19.65.190:37367)
0903 14:27:22.797840 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:22.797843 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:22.798366 (+ 523us) server_negotiation.cc:408] Connection header received
0903 14:27:22.799040 (+ 674us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:22.799043 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:22.799097 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:22.799163 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:22.800531 (+ 1368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.800970 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.801592 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.801758 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.803337 (+ 1579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:22.803364 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:22.803366 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:22.803388 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:22.804486 (+ 1098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.804905 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.804909 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.804910 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.804954 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:22.805204 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:22.805207 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:22.805209 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:22.805339 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:22.805459 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:22.805879 (+ 420us) server_negotiation.cc:300] Negotiation successful
0903 14:27:22.805980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:22.806305 19718 external_mini_cluster.cc:1468] Setting key f9eadc6c915ebaadc41483c42f51ade2
I20250903 14:27:22.806572 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.797905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:49449)
0903 14:27:22.798240 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:22.798270 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:22.798880 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:22.799293 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:22.799299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:22.799574 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:22.800400 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.800412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.801071 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:22.801074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:22.801425 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.801432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:22.801648 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:22.802067 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:22.802085 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:22.803218 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:22.804592 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.804597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.804606 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.804812 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.805032 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:22.805037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:22.805039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:22.805120 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:22.805469 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:22.805476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:22.805662 (+ 186us) client_negotiation.cc:769] Sending connection context
0903 14:27:22.805828 (+ 166us) client_negotiation.cc:240] Negotiation successful
0903 14:27:22.806032 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20250903 14:27:22.807324 26920 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:22.807524 26920 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:22.807974 26920 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
I20250903 14:27:22.808750 26455 ts_manager.cc:194] Registered new tserver with Master: f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:22.809201 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:49449
I20250903 14:27:22.819253 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.65.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:22.856920 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:36775
--webserver_interface=127.19.65.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:37367,127.19.65.189:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:22.945235 26927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:22.945410 26927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:22.945470 26927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:22.948045 26927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:22.948087 26927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:22.948105 26927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:22.948127 26927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:22.948160 26927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:22.951551 26927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:37367,127.19.65.189:36775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.65.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
W20250903 14:27:22.951679 26927 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.65.190:37367,127.19.65.189:36775'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20250903 14:27:22.952306 26927 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:22.952948 26927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:22.957636 26935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.957645 26932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:22.957722 26933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:22.957749 26927 server_base.cc:1047] running on GCE node
I20250903 14:27:22.958261 26927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:22.958940 26927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:22.960070 26927 hybrid_clock.cc:648] HybridClock initialized: now 1756909642960052 us; error 31 us; skew 500 ppm
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:22.962301 26927 init.cc:377] Logged in from keytab as oryx/127.19.65.189@KRBTEST.COM (short username oryx)
I20250903 14:27:22.963101 26927 webserver.cc:480] Webserver started at http://127.19.65.189:40579/ using document root <none> and password file <none>
I20250903 14:27:22.963557 26927 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:22.963599 26927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:22.963748 26927 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:22.965190 26927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/instance:
uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "82a158a37c64ce8c3cbf696706762559"
server_key_iv: "c215a03ddd5606a6ed519f2c6f65ce4c"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.965674 26927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal/instance:
uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "82a158a37c64ce8c3cbf696706762559"
server_key_iv: "c215a03ddd5606a6ed519f2c6f65ce4c"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.968295 26927 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:22.970036 26942 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:22.970886 26927 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:22.970978 26927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "82a158a37c64ce8c3cbf696706762559"
server_key_iv: "c215a03ddd5606a6ed519f2c6f65ce4c"
server_key_version: "encryptionkey@0"
I20250903 14:27:22.971048 26927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:22.981873 26927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:22.984153 26927 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:22.984313 26927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:22.990003 26994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.189:36775 every 8 connection(s)
I20250903 14:27:22.990012 26927 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:36775
I20250903 14:27:22.990805 26927 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/info.pb
I20250903 14:27:22.992810 26995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:22.996001 26995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:27:23.001019 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 26927
Sep 03 14:27:22 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:23.005836 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.997305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.19.65.190:37367)
0903 14:27:22.997433 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:22.997437 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:22.998030 (+ 593us) server_negotiation.cc:408] Connection header received
0903 14:27:22.998566 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:22.998569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:22.998614 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:22.998678 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:22.999630 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.000068 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.000578 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.000683 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.002597 (+ 1914us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.002614 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.002617 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.002643 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.003946 (+ 1303us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.004389 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.004393 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.004395 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.004441 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.004760 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.004764 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.004765 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.004976 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.005087 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.005532 (+ 445us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.005645 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:27:23.006184 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:22.997561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37112)
0903 14:27:22.997911 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:22.997939 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:22.998427 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:22.998801 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:22.998811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:22.999071 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:22.999522 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:22.999532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.000151 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.000153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.000494 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.000500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.000628 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.001090 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.001118 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.002479 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.004035 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.004040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.004048 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.004288 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.004524 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.004529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.004531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.004646 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.005080 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.005084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.005264 (+ 180us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.005413 (+ 149us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.005627 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20250903 14:27:23.007308 26995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:27:23.010002 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.001638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.19.65.189:36775)
0903 14:27:23.001881 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.001887 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.002060 (+ 173us) server_negotiation.cc:408] Connection header received
0903 14:27:23.002212 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.002217 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.002325 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.002459 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.003187 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.003776 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.004388 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.004615 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.006860 (+ 2245us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.006877 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.006885 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.006907 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.008745 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.009060 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.009064 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.009065 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.009101 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.009278 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.009280 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.009281 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.009402 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.009601 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.009754 (+ 153us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.009870 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20250903 14:27:23.014031 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.007693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34584)
0903 14:27:23.007773 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.007786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.007852 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.008126 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.008131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.008285 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.008455 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.008462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.008998 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.009003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.009357 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.009364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.009435 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.009854 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.009869 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.010938 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.012988 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.012991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.012993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.013151 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.013330 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.013333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.013337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.013377 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.013656 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.013659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.013718 (+ 59us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.013795 (+ 77us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.013900 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20250903 14:27:23.014083 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.007738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.19.65.189:36775)
0903 14:27:23.007908 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.007910 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.007920 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:23.007943 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.007947 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.007987 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.008079 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.008546 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.008915 (+ 369us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.009444 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.009899 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.011034 (+ 1135us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.011045 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.011047 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.011066 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.012895 (+ 1829us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.013226 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.013229 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.013232 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.013264 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.013442 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.013444 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.013446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.013578 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.013656 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.013837 (+ 181us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.013952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20250903 14:27:23.016026 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Bootstrap starting.
I20250903 14:27:23.017725 26995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:23.018271 26995 log.cc:826] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:23.019491 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: No bootstrap required, opened a new log
I20250903 14:27:23.021261 26995 raft_consensus.cc:357] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } }
I20250903 14:27:23.021385 26995 raft_consensus.cc:383] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:23.021436 26995 raft_consensus.cc:738] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a76ae20d7c904e3ea2ebe7e94df5dfae, State: Initialized, Role: FOLLOWER
I20250903 14:27:23.021806 26995 consensus_queue.cc:260] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } }
I20250903 14:27:23.022145 27002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } } }
I20250903 14:27:23.022228 27002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:23.022595 26995 sys_catalog.cc:564] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: configured and running, proceeding with master startup.
W20250903 14:27:23.026156 27013 catalog_manager.cc:1560] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:27:23.026216 27013 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:27:23.027669 26927 master_runner.cc:428] Detected that this master a76ae20d7c904e3ea2ebe7e94df5dfae is joining an existing cluster
I20250903 14:27:23.027722 26927 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.65.189:36775
I20250903 14:27:23.035604 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.029238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.19.65.190:37367)
0903 14:27:23.029326 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.029329 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.029516 (+ 187us) server_negotiation.cc:408] Connection header received
0903 14:27:23.029648 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.029651 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.029694 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.029743 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.030410 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031079 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.032153 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.032336 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.033054 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.033065 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.033067 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.033095 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.034340 (+ 1245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.034749 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.034751 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.034753 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.034788 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.034992 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.034994 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.034995 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.035104 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.035171 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.035421 (+ 250us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.035515 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20250903 14:27:23.035657 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.029135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37116)
0903 14:27:23.029390 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.029404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.029536 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.029860 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.029862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.030047 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.030289 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.030295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031199 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.031203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.032044 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.032053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.032147 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.032647 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.032662 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.032951 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.034447 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.034451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.034453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.034658 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.034860 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.034863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.034865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.034917 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.035182 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.035185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.035274 (+ 89us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.035392 (+ 118us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.035520 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20250903 14:27:23.052281 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.029586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.19.65.189:36775)
0903 14:27:23.029663 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.029667 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.029837 (+ 170us) server_negotiation.cc:408] Connection header received
0903 14:27:23.029925 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.029928 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.029957 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.030013 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.030451 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031019 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.031553 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031727 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.032346 (+ 619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.032359 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.032361 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.032379 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.051007 (+ 18628us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051433 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.051436 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.051437 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.051471 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051676 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.051679 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.051681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.051806 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.051885 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.052072 (+ 187us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.052184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":23}
I20250903 14:27:23.052291 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.029589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34592)
0903 14:27:23.029763 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.029773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.029850 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.030031 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.030033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.030195 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.030345 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.030352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031144 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.031149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.031458 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.031464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.031557 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.031979 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.031988 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.032244 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.051100 (+ 18856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.051104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.051358 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.051554 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.051557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.051600 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.051876 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.051879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.051950 (+ 71us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.052030 (+ 80us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.052136 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20250903 14:27:23.052819 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.038940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.19.65.189:36775)
0903 14:27:23.039037 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.039040 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.039315 (+ 275us) server_negotiation.cc:408] Connection header received
0903 14:27:23.039474 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.039478 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.039526 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.039599 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.040122 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.040704 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.041141 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.041298 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.042829 (+ 1531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.042843 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.042846 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.042868 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.051425 (+ 8557us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051829 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.051832 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.051833 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.051872 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.052088 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.052091 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.052092 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.052209 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.052263 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.052616 (+ 353us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.052703 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:27:23.053244 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.038884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34596)
0903 14:27:23.039207 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.039235 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.039360 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.039613 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.039616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.039870 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.040029 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.040034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.040782 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.040785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.041063 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.041067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.041138 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.041553 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.041564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.042730 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.051515 (+ 8785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.051529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.051753 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.051941 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.051944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.051946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.051996 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.052332 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.052338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.052435 (+ 97us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.052959 (+ 524us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.053107 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20250903 14:27:23.054224 26455 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.19.65.189:36775: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
member_type: NON_VOTER
last_known_addr {
host: "127.19.65.189"
port: 36775
}
attrs {
promote: true
}
}
dest_uuid: "55f880c1f5e3421ab6ec375c6d4932d1"
cas_config_opid_index: -1
I20250903 14:27:23.054795 26455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } }
I20250903 14:27:23.056165 27023 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } }
I20250903 14:27:23.056684 26969 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:23.057621 27023 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.19.65.189:36775
I20250903 14:27:23.057780 27022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } }
I20250903 14:27:23.057885 27022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:23.058038 26927 master.cc:561] Master@127.19.65.189:36775 shutting down...
I20250903 14:27:23.058063 26969 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Refusing update from remote peer 55f880c1f5e3421ab6ec375c6d4932d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250903 14:27:23.058781 27022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 14:27:23.060578 26440 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20250903 14:27:23.061331 26440 consensus_peers.cc:489] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 -> Peer a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189:36775): Couldn't send request to peer a76ae20d7c904e3ea2ebe7e94df5dfae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250903 14:27:23.061682 26927 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:23.061887 26927 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:23.061983 26927 tablet_replica.cc:331] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: stopping tablet replica
W20250903 14:27:23.064442 26405 connection.cc:537] client connection to 127.19.65.189:36775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:27:23.075708 26927 master.cc:583] Master@127.19.65.189:36775 shutdown complete.
I20250903 14:27:23.075757 26927 master_runner.cc:305] Clearing existing system tablet
I20250903 14:27:23.075963 26927 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:23.076009 26927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:23.077031 26927 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250903 14:27:23.077796 27027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:23.077999 26927 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:23.078090 26927 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:23.078121 26927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "82a158a37c64ce8c3cbf696706762559"
server_key_iv: "c215a03ddd5606a6ed519f2c6f65ce4c"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.079394 26927 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:23.080739 26927 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250903 14:27:23.080796 26927 log.cc:1199] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20250903 14:27:23.080965 26927 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Deleting consensus metadata
I20250903 14:27:23.081085 26927 master_runner.cc:315] Copying system tablet from 127.19.65.190:37367
I20250903 14:27:23.081920 26927 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: tablet copy: Beginning tablet copy session from remote peer at address 127.19.65.190:37367
I20250903 14:27:23.087695 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.082370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.19.65.190:37367)
0903 14:27:23.082476 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.082479 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.082562 (+ 83us) server_negotiation.cc:408] Connection header received
0903 14:27:23.082657 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.082660 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.082693 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.082752 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.083276 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.083676 (+ 400us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.084397 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.084510 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.085362 (+ 852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.085377 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.085379 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.085399 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.086630 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.086985 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.086987 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.086988 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.087017 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.087182 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.087183 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.087185 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.087282 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.087339 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.087542 (+ 203us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.087610 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:27:23.087751 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.082272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37130)
0903 14:27:23.082488 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.082501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.082590 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.082786 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.082789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.082954 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.083194 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.083200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.083755 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.083758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.084266 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.084276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.084420 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.084889 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.084904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.085243 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.086721 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.086724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.086726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.086912 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.087075 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.087078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.087080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.087123 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.087342 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.087345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.087426 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.087516 (+ 90us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.087615 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20250903 14:27:23.088158 26475 tablet_copy_service.cc:140] P 55f880c1f5e3421ab6ec375c6d4932d1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a76ae20d7c904e3ea2ebe7e94df5dfae ({username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:37130)
I20250903 14:27:23.088279 26475 tablet_copy_service.cc:161] P 55f880c1f5e3421ab6ec375c6d4932d1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a76ae20d7c904e3ea2ebe7e94df5dfae at {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:37130: session id = a76ae20d7c904e3ea2ebe7e94df5dfae-00000000000000000000000000000000
I20250903 14:27:23.089493 26475 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 14:27:23.090688 26927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250903 14:27:23.091897 26927 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: tablet copy: Starting download of 0 data blocks...
I20250903 14:27:23.092022 26927 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: tablet copy: Starting download of 1 WAL segments...
I20250903 14:27:23.093396 26927 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 14:27:23.094668 26475 tablet_copy_service.cc:342] P 55f880c1f5e3421ab6ec375c6d4932d1: Request end of tablet copy session a76ae20d7c904e3ea2ebe7e94df5dfae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:37130
I20250903 14:27:23.094767 26475 tablet_copy_service.cc:434] P 55f880c1f5e3421ab6ec375c6d4932d1: ending tablet copy session a76ae20d7c904e3ea2ebe7e94df5dfae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a76ae20d7c904e3ea2ebe7e94df5dfae
W20250903 14:27:23.098327 26927 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250903 14:27:23.099881 26927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:23.101329 27034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:23.101351 27035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:23.101449 27037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:23.101749 26927 server_base.cc:1047] running on GCE node
I20250903 14:27:23.101856 26927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:23.102021 26927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:23.103147 26927 hybrid_clock.cc:648] HybridClock initialized: now 1756909643103122 us; error 36 us; skew 500 ppm
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:23.104427 26927 init.cc:377] Logged in from keytab as oryx/127.19.65.189@KRBTEST.COM (short username oryx)
I20250903 14:27:23.104820 26927 webserver.cc:480] Webserver started at http://127.19.65.189:36251/ using document root <none> and password file <none>
I20250903 14:27:23.104970 26927 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:23.105007 26927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:23.105962 26927 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:23.106724 27044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:23.106954 26927 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:23.107019 26927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "82a158a37c64ce8c3cbf696706762559"
server_key_iv: "c215a03ddd5606a6ed519f2c6f65ce4c"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.107084 26927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:23.130254 26927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:23.131740 26927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:23.136125 26927 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:36775
I20250903 14:27:23.136160 27096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.189:36775 every 8 connection(s)
I20250903 14:27:23.136577 26927 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-1/data/info.pb
I20250903 14:27:23.137724 27097 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:27:23.138374 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Bootstrap starting.
I20250903 14:27:23.144940 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:27:23.145174 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Bootstrap complete.
I20250903 14:27:23.145634 27097 raft_consensus.cc:357] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } }
I20250903 14:27:23.145738 27097 raft_consensus.cc:738] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a76ae20d7c904e3ea2ebe7e94df5dfae, State: Initialized, Role: LEARNER
I20250903 14:27:23.145845 27097 consensus_queue.cc:260] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } }
I20250903 14:27:23.146106 27100 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } }
I20250903 14:27:23.146190 27100 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: LEARNER
I20250903 14:27:23.146238 27097 sys_catalog.cc:564] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:23.148391 27111 catalog_manager.cc:1261] Loaded cluster ID: 8be42f9d5dde470aa2cf5c0895508058
I20250903 14:27:23.148435 27111 catalog_manager.cc:1554] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: loading cluster ID for follower catalog manager: success
I20250903 14:27:23.149812 27111 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: acquiring CA information for follower catalog manager: success
I20250903 14:27:23.150761 27111 catalog_manager.cc:1604] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:27:23.172004 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.165740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.19.65.189:36775)
0903 14:27:23.165934 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.165938 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.165952 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:23.165987 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.165990 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.166027 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.166109 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.166722 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.167790 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.168303 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.168481 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.169275 (+ 794us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.169290 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.169293 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.169314 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.170825 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.171146 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.171149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.171151 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.171200 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.171361 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.171366 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.171368 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.171518 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.171630 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.171756 (+ 126us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.171854 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.19.65.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:23.208823 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:46023
--webserver_interface=127.19.65.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:23.301270 27117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:23.301425 27117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:23.301504 27117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:23.304209 27117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:23.304257 27117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:23.304277 27117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:23.304312 27117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:23.304329 27117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:23.308053 27117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.65.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:23.308830 27117 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:23.309526 27117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:23.313966 27122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:23.314069 27117 server_base.cc:1047] running on GCE node
W20250903 14:27:23.313982 27125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:23.313992 27123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:23.314617 27117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:23.315347 27117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:23.316495 27117 hybrid_clock.cc:648] HybridClock initialized: now 1756909643316485 us; error 28 us; skew 500 ppm
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:23.318733 27117 init.cc:377] Logged in from keytab as oryx/127.19.65.188@KRBTEST.COM (short username oryx)
I20250903 14:27:23.319525 27117 webserver.cc:480] Webserver started at http://127.19.65.188:34953/ using document root <none> and password file <none>
I20250903 14:27:23.319931 27117 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:23.319969 27117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:23.320092 27117 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:23.321465 27117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/instance:
uuid: "9f0c7257d43944b6b75d66a871ac979e"
format_stamp: "Formatted at 2025-09-03 14:27:23 on dist-test-slave-4wpb"
server_key: "7ed862908e72cfccb5ee5ac2d8152e84"
server_key_iv: "66f6aadab321855b469b135866a634d1"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.321916 27117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal/instance:
uuid: "9f0c7257d43944b6b75d66a871ac979e"
format_stamp: "Formatted at 2025-09-03 14:27:23 on dist-test-slave-4wpb"
server_key: "7ed862908e72cfccb5ee5ac2d8152e84"
server_key_iv: "66f6aadab321855b469b135866a634d1"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.324630 27117 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:23.326436 27132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:23.327279 27117 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:23.327359 27117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
uuid: "9f0c7257d43944b6b75d66a871ac979e"
format_stamp: "Formatted at 2025-09-03 14:27:23 on dist-test-slave-4wpb"
server_key: "7ed862908e72cfccb5ee5ac2d8152e84"
server_key_iv: "66f6aadab321855b469b135866a634d1"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.327421 27117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:23.339908 27117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:23.342000 27117 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:23.342119 27117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:23.348133 27117 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:46023
I20250903 14:27:23.348140 27184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.188:46023 every 8 connection(s)
I20250903 14:27:23.348875 27117 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/info.pb
I20250903 14:27:23.350852 27185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:23.353798 27185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:27:23.356601 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27117
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:23.364619 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.355210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.19.65.190:37367)
0903 14:27:23.355337 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.355340 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.355843 (+ 503us) server_negotiation.cc:408] Connection header received
0903 14:27:23.356381 (+ 538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.356384 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.356437 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.356519 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.357719 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.358393 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.359004 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.359132 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.361087 (+ 1955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.361105 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.361108 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.361128 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.362570 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.363074 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.363078 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.363081 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.363135 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.363564 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.363570 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.363574 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.363781 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.363905 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.364316 (+ 411us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.364444 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 14:27:23.365130 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.355354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37140)
0903 14:27:23.355727 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.355754 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.356226 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.356624 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.356630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.356915 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.357591 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.357605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.358493 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.358495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.358905 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.358912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.359060 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.359534 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.359566 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.360962 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.362651 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.362660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.362667 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.362979 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.363224 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.363229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.363232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.363370 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.363891 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.363896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.364077 (+ 181us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.364223 (+ 146us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.364430 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20250903 14:27:23.366182 27185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:27:23.366374 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.357290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.19.65.188:46023)
0903 14:27:23.357564 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.357569 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.357588 (+ 19us) server_negotiation.cc:408] Connection header received
0903 14:27:23.357620 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.357624 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.357753 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.357849 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.358662 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.359473 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.360004 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.360137 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.362399 (+ 2262us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.362412 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.362418 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.362436 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.364566 (+ 2130us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.365000 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.365004 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.365005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.365048 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.365317 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.365321 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.365322 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.365573 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.365705 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.366162 (+ 457us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.366256 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20250903 14:27:23.372175 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.366510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34618)
0903 14:27:23.366591 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.366604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.366666 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.367020 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.367023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.367156 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.367315 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.367321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.367926 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.367928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.368233 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.368239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.368313 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.368665 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.368674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.369597 (+ 923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.371263 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.371266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.371267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.371422 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.371612 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.371614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.371615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.371646 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.371861 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.371863 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.371918 (+ 55us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.371989 (+ 71us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.372074 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 14:27:23.372236 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.366739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.19.65.189:36775)
0903 14:27:23.366852 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.366855 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.366865 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:23.366896 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.366898 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.366933 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.366989 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.367401 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.367852 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.368310 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.368463 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.369689 (+ 1226us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.369703 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.369705 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.369725 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.371043 (+ 1318us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.371492 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.371494 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.371496 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.371550 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.371699 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.371701 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.371702 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.371806 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.371868 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.372019 (+ 151us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.372119 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:27:23.372792 27185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:23.378361 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.373159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36456)
0903 14:27:23.373250 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.373263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.373330 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.373529 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.373532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.373656 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.373766 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.373771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.374324 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.374327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.374619 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.374624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.374719 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.375105 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.375116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.376063 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.377561 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.377563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.377564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.377696 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.377852 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.377854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.377855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.377886 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.378077 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.378079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.378153 (+ 74us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.378213 (+ 60us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.378284 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250903 14:27:23.378525 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.373234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.19.65.188:46023)
0903 14:27:23.373322 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.373325 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.373336 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:23.373407 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.373410 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.373440 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.373525 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.373856 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.374212 (+ 356us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.374712 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.374905 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.376154 (+ 1249us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.376166 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.376168 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.376185 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.377432 (+ 1247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.377764 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.377766 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.377767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.377798 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.377936 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.377937 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.377938 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.378026 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.378082 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.378235 (+ 153us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.378329 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":34}
I20250903 14:27:23.380386 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Bootstrap starting.
I20250903 14:27:23.382239 27185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:23.382797 27185 log.cc:826] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:23.384152 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: No bootstrap required, opened a new log
I20250903 14:27:23.386247 27185 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } }
I20250903 14:27:23.386371 27185 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:23.386422 27185 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0c7257d43944b6b75d66a871ac979e, State: Initialized, Role: FOLLOWER
I20250903 14:27:23.386746 27185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } }
I20250903 14:27:23.387084 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } } }
I20250903 14:27:23.387171 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:23.387542 27185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: configured and running, proceeding with master startup.
W20250903 14:27:23.391005 27201 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:27:23.391059 27201 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:27:23.393584 27117 master_runner.cc:428] Detected that this master 9f0c7257d43944b6b75d66a871ac979e is joining an existing cluster
I20250903 14:27:23.393636 27117 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.65.188:46023
I20250903 14:27:23.416410 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.396456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.19.65.188:46023)
0903 14:27:23.396546 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.396549 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.396687 (+ 138us) server_negotiation.cc:408] Connection header received
0903 14:27:23.396770 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.396773 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.396801 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.396852 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.410163 (+ 13311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.410749 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.411437 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411580 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.412914 (+ 1334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.412927 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.412930 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.412961 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.414929 (+ 1968us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415505 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.415508 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.415510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.415544 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415730 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.415733 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.415734 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.415861 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.415919 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.416179 (+ 260us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.416270 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32,"spinlock_wait_cycles":6400}
I20250903 14:27:23.416646 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.396023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.19.65.189:36775)
0903 14:27:23.396125 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.396129 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.396340 (+ 211us) server_negotiation.cc:408] Connection header received
0903 14:27:23.396487 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.396491 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.396527 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.396567 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.410163 (+ 13596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.410837 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.411607 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411774 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.413108 (+ 1334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.413124 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.413127 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.413149 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.414991 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415543 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.415549 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.415551 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.415596 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415853 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.415856 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.415857 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.416009 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.416072 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.416401 (+ 329us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.416500 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:23.416791 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.396211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36466)
0903 14:27:23.396622 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.396630 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.396705 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.396869 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.396872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.409720 (+ 12848us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.410014 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.410022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.410869 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.410871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.411346 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.411354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411661 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.412263 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.412281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.412794 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.415049 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.415055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.415380 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.415621 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.415626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.415662 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.415966 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.415968 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.416028 (+ 60us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.416545 (+ 517us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.416639 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"spinlock_wait_cycles":21376,"thread_start_us":71,"threads_started":1}
I20250903 14:27:23.417299 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.395950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34628)
0903 14:27:23.396261 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.396273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.396398 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.396666 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.396668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.409791 (+ 13123us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.410041 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.410048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.410958 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.410961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.411492 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.411500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.412138 (+ 638us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.412490 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.412502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.412976 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.415117 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.415123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.415405 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.415698 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.415704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.415760 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.416144 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.416147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.416221 (+ 74us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.417018 (+ 797us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.417156 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":11520,"thread_start_us":77,"threads_started":1}
I20250903 14:27:23.417436 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.395682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.19.65.190:37367)
0903 14:27:23.395792 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.395796 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.395989 (+ 193us) server_negotiation.cc:408] Connection header received
0903 14:27:23.396172 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.396175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.396217 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.396267 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.410533 (+ 14266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411197 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.411947 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.412138 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.413264 (+ 1126us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.413281 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.413287 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.413314 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.415733 (+ 2419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.416116 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.416119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.416121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.416164 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.416412 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.416415 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.416417 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.416559 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.416620 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.417189 (+ 569us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.417277 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:27:23.418076 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.395616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37146)
0903 14:27:23.395922 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.395935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.396051 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.396412 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.396417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.409735 (+ 13318us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.410420 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.410427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411316 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.411319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.411842 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.411849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.411949 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.412295 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.412326 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.413138 (+ 812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.415841 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.415844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.415846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.416009 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.416258 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.416261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.416262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.416309 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.416923 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.416926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.417001 (+ 75us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.417830 (+ 829us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.417931 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":73,"threads_started":1}
I20250903 14:27:23.421401 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.419340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37154)
0903 14:27:23.419434 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.419446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.419507 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.419718 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.419720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.419745 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:23.419866 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.419870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.420403 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.420405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.421000 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.421007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.421112 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.421167 (+ 55us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.421216 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.421286 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"spinlock_wait_cycles":384}
I20250903 14:27:23.421859 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.419469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.19.65.190:37367)
0903 14:27:23.419566 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.419568 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.419577 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:27:23.419610 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.419612 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.419640 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.419701 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:23.419949 (+ 248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.420313 (+ 364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.421092 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.421633 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.421690 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.421731 (+ 41us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.421781 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909642, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:23.425568 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.422814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34642)
0903 14:27:23.422945 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.422956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.423019 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.423748 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.423750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.423764 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:23.423910 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.423914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.424554 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.424558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.425145 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.425154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.425263 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.425283 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.425338 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.425413 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20250903 14:27:23.426062 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.422883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.19.65.189:36775)
0903 14:27:23.423488 (+ 605us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.423493 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.423505 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:23.423540 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.423545 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.423598 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.423679 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:23.424010 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.424447 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.425267 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.425747 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.425882 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.425925 (+ 43us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.425969 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521}
I20250903 14:27:23.428153 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.421123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36468)
0903 14:27:23.421552 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.421566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.421638 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.421867 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.421872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.422023 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.422188 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.422197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.422868 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.422871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.423196 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.423203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.423283 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.423699 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.423716 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.425042 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.427141 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.427144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.427146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.427286 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.427514 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.427517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.427518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.427561 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.427857 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.427859 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.427914 (+ 55us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.427979 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.428053 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":82,"threads_started":1}
I20250903 14:27:23.428231 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.421411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.19.65.188:46023)
0903 14:27:23.421507 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.421511 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.421618 (+ 107us) server_negotiation.cc:408] Connection header received
0903 14:27:23.421729 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.421732 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.421773 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.421844 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.422298 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.422755 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.423290 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.423461 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.425162 (+ 1701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.425178 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.425184 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.425214 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.427049 (+ 1835us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.427381 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.427384 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.427386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.427425 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.427636 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.427639 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.427641 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.427778 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.427870 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.428029 (+ 159us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.428116 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20250903 14:27:23.428911 26454 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.19.65.188:46023: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e"
member_type: NON_VOTER
last_known_addr {
host: "127.19.65.188"
port: 46023
}
attrs {
promote: true
}
}
dest_uuid: "55f880c1f5e3421ab6ec375c6d4932d1"
cas_config_opid_index: 5
I20250903 14:27:23.429307 26454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } }
I20250903 14:27:23.430130 27022 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.431075 27022 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.19.65.188:46023
I20250903 14:27:23.431223 27022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.431293 27022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:23.431658 27117 master.cc:561] Master@127.19.65.188:46023 shutting down...
I20250903 14:27:23.433058 27071 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 LEARNER]: Refusing update from remote peer 55f880c1f5e3421ab6ec375c6d4932d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250903 14:27:23.433530 27023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250903 14:27:23.435246 27071 raft_consensus.cc:2953] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.436733 27212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.436825 27212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: LEARNER
I20250903 14:27:23.437995 27100 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f880c1f5e3421ab6ec375c6d4932d1. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: true } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.438083 27100 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: LEARNER
I20250903 14:27:23.438815 27117 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:23.439015 27117 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 14:27:23.439057 26440 consensus_peers.cc:489] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 -> Peer 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023): Couldn't send request to peer 9f0c7257d43944b6b75d66a871ac979e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250903 14:27:23.439090 27117 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: stopping tablet replica
I20250903 14:27:23.440063 27023 raft_consensus.cc:1062] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: attempting to promote NON_VOTER a76ae20d7c904e3ea2ebe7e94df5dfae to VOTER
I20250903 14:27:23.440362 27023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } }
W20250903 14:27:23.441403 26440 connection.cc:537] client connection to 127.19.65.188:46023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:27:23.441907 27071 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 LEARNER]: Refusing update from remote peer 55f880c1f5e3421ab6ec375c6d4932d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250903 14:27:23.442222 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.442000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36482)
0903 14:27:23.442122 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.442169 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20250903 14:27:23.442260 27022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250903 14:27:23.442350 26440 consensus_peers.cc:489] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 -> Peer 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023): Couldn't send request to peer 9f0c7257d43944b6b75d66a871ac979e. Status: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:27:23.443610 27213 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.444202 27071 raft_consensus.cc:2953] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.444867 27023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.444955 27023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:23.445111 27023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.445173 27023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:23.445399 27100 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.445535 27100 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:23.445824 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.445664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36484)
0903 14:27:23.445737 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.445771 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20250903 14:27:23.445987 26455 master.cc:679] Network error: unable to get registration information for peer 9f0c7257d43944b6b75d66a871ac979e: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
I20250903 14:27:23.452766 26648 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:23.453342 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.452951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36494)
0903 14:27:23.453234 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.453279 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
W20250903 14:27:23.453524 26455 master.cc:679] Network error: unable to get registration information for peer 9f0c7257d43944b6b75d66a871ac979e: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
I20250903 14:27:23.453778 27117 master.cc:583] Master@127.19.65.188:46023 shutdown complete.
I20250903 14:27:23.453819 27117 master_runner.cc:305] Clearing existing system tablet
I20250903 14:27:23.454003 27117 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:23.454048 27117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:23.455191 27117 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:23.455977 27215 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:23.456189 27117 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:23.456290 27117 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:23.456316 27117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
uuid: "9f0c7257d43944b6b75d66a871ac979e"
format_stamp: "Formatted at 2025-09-03 14:27:23 on dist-test-slave-4wpb"
server_key: "7ed862908e72cfccb5ee5ac2d8152e84"
server_key_iv: "66f6aadab321855b469b135866a634d1"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.457861 27117 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:23.459651 27117 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250903 14:27:23.459722 27117 log.cc:1199] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20250903 14:27:23.459918 27117 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Deleting consensus metadata
I20250903 14:27:23.460045 27117 master_runner.cc:315] Copying system tablet from 127.19.65.190:37367
I20250903 14:27:23.461004 27117 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.65.190:37367
I20250903 14:27:23.463793 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.463625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36498)
0903 14:27:23.463713 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.463749 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20250903 14:27:23.463956 26455 master.cc:679] Network error: unable to get registration information for peer 9f0c7257d43944b6b75d66a871ac979e: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
I20250903 14:27:23.467317 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.461340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37164)
0903 14:27:23.461611 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.461623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.461707 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.461882 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.461884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.462059 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.462288 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.462295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.463176 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.463180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.463697 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.463705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.464058 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.464570 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.464586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.464842 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.466443 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.466446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.466447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.466579 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.466764 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.466766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.466767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.466797 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.467028 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.467029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.467083 (+ 54us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.467148 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.467222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20250903 14:27:23.467403 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.461433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.19.65.190:37367)
0903 14:27:23.461584 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.461587 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.461678 (+ 91us) server_negotiation.cc:408] Connection header received
0903 14:27:23.461770 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.461772 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.461802 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.461855 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.462417 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.463056 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.463811 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.463970 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.464954 (+ 984us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.464969 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.464973 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.464998 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.466339 (+ 1341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.466654 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.466657 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.466659 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.466700 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.466841 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.466844 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.466846 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.466968 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.467041 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.467199 (+ 158us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.467284 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20250903 14:27:23.467835 26475 tablet_copy_service.cc:140] P 55f880c1f5e3421ab6ec375c6d4932d1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9f0c7257d43944b6b75d66a871ac979e ({username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:37164)
I20250903 14:27:23.467900 26475 tablet_copy_service.cc:161] P 55f880c1f5e3421ab6ec375c6d4932d1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9f0c7257d43944b6b75d66a871ac979e at {username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:37164: session id = 9f0c7257d43944b6b75d66a871ac979e-00000000000000000000000000000000
I20250903 14:27:23.468658 26475 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 14:27:23.469578 27117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250903 14:27:23.471199 27117 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: tablet copy: Starting download of 0 data blocks...
I20250903 14:27:23.471349 27117 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: tablet copy: Starting download of 1 WAL segments...
I20250903 14:27:23.472543 27117 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 14:27:23.473729 26475 tablet_copy_service.cc:342] P 55f880c1f5e3421ab6ec375c6d4932d1: Request end of tablet copy session 9f0c7257d43944b6b75d66a871ac979e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:37164
I20250903 14:27:23.473803 26475 tablet_copy_service.cc:434] P 55f880c1f5e3421ab6ec375c6d4932d1: ending tablet copy session 9f0c7257d43944b6b75d66a871ac979e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9f0c7257d43944b6b75d66a871ac979e
W20250903 14:27:23.477336 27117 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250903 14:27:23.479054 27117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:23.480324 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:23.480535 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:23.480758 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:23.481632 27117 server_base.cc:1047] running on GCE node
I20250903 14:27:23.481757 27117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:23.481968 27117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:23.482967 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.482792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36510)
0903 14:27:23.482880 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.482926 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20250903 14:27:23.483101 27117 hybrid_clock.cc:648] HybridClock initialized: now 1756909643483078 us; error 25 us; skew 500 ppm
W20250903 14:27:23.483170 26455 master.cc:679] Network error: unable to get registration information for peer 9f0c7257d43944b6b75d66a871ac979e: Client connection negotiation failed: client connection to 127.19.65.188:46023: connect: Connection refused (error 111)
Sep 03 14:27:23 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909643, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:23.484563 27117 init.cc:377] Logged in from keytab as oryx/127.19.65.188@KRBTEST.COM (short username oryx)
I20250903 14:27:23.484975 27117 webserver.cc:480] Webserver started at http://127.19.65.188:45957/ using document root <none> and password file <none>
I20250903 14:27:23.485146 27117 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:23.485183 27117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:23.486083 27117 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:23.486768 27232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:23.486956 27117 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:23.487015 27117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
uuid: "9f0c7257d43944b6b75d66a871ac979e"
format_stamp: "Formatted at 2025-09-03 14:27:23 on dist-test-slave-4wpb"
server_key: "7ed862908e72cfccb5ee5ac2d8152e84"
server_key_iv: "66f6aadab321855b469b135866a634d1"
server_key_version: "encryptionkey@0"
I20250903 14:27:23.487071 27117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:23.500809 27117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:23.502171 27117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:23.506142 27117 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:46023
I20250903 14:27:23.506198 27284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.188:46023 every 8 connection(s)
I20250903 14:27:23.506569 27117 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/master-2/data/info.pb
I20250903 14:27:23.507881 27285 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:27:23.508710 27285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Bootstrap starting.
I20250903 14:27:23.518589 27285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:27:23.518868 27285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: Bootstrap complete.
I20250903 14:27:23.519413 27285 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } }
I20250903 14:27:23.519536 27285 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f0c7257d43944b6b75d66a871ac979e, State: Initialized, Role: LEARNER
I20250903 14:27:23.519634 27285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } }
I20250903 14:27:23.519946 27289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.520058 27285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:23.520123 27289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: LEARNER
I20250903 14:27:23.524454 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.517674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36526)
0903 14:27:23.517767 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:23.517780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:23.517847 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:23.518206 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:23.518208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:23.518350 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:23.518513 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.518518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.519387 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.519392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:23.519813 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:23.519824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.519935 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.520437 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:23.520455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:23.520733 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:23.522632 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.522636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.522638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.522836 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.523112 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:23.523115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:23.523117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:23.523164 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:23.523945 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:23.523950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:23.524043 (+ 93us) client_negotiation.cc:769] Sending connection context
0903 14:27:23.524141 (+ 98us) client_negotiation.cc:240] Negotiation successful
0903 14:27:23.524264 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 14:27:23.524626 27300 catalog_manager.cc:1261] Loaded cluster ID: 8be42f9d5dde470aa2cf5c0895508058
I20250903 14:27:23.524706 27300 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: loading cluster ID for follower catalog manager: success
I20250903 14:27:23.524968 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:23.517825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.19.65.188:46023)
0903 14:27:23.518012 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:23.518015 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:23.518023 (+ 8us) server_negotiation.cc:408] Connection header received
0903 14:27:23.518052 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:23.518054 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:23.518093 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:23.518174 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:23.518629 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.519291 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:23.520132 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:23.520296 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:23.520853 (+ 557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:23.520868 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:23.520871 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:23.520894 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:23.522504 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.522972 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.522974 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.522976 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.523014 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:23.523259 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:23.523264 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:23.523267 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:23.523403 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:23.524638 (+ 1235us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:23.524718 (+ 80us) server_negotiation.cc:300] Negotiation successful
0903 14:27:23.524826 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20250903 14:27:23.527261 27300 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: acquiring CA information for follower catalog manager: success
I20250903 14:27:23.528203 27300 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:27:23.636132 26784 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:23.810766 26920 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:23.883244 27259 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250903 14:27:23.883654 27289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f880c1f5e3421ab6ec375c6d4932d1. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:23.883759 27289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: LEARNER
I20250903 14:27:24.390658 27213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: true } } }
I20250903 14:27:24.390808 27213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:24.391595 27213 raft_consensus.cc:1062] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: attempting to promote NON_VOTER 9f0c7257d43944b6b75d66a871ac979e to VOTER
I20250903 14:27:24.391903 27213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:24.392908 27259 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 LEARNER]: Refusing update from remote peer 55f880c1f5e3421ab6ec375c6d4932d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250903 14:27:24.392933 27071 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Refusing update from remote peer 55f880c1f5e3421ab6ec375c6d4932d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250903 14:27:24.393311 27302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250903 14:27:24.393553 27213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250903 14:27:24.394992 27302 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.395462 27071 raft_consensus.cc:2953] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.396275 27213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.396255 27302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.396363 27213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:24.396400 27302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:24.396553 27305 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.396642 27305 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:24.397059 27213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.397141 27213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:24.397434 27259 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.399320 27306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:24.399418 27306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:24.503320 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.501247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37168)
0903 14:27:24.501520 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.501531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.501598 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.501781 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.501783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.501795 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.501974 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.501978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.502560 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.502562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.503051 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.503057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.503132 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.503149 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.503181 (+ 32us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.503228 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20250903 14:27:24.503625 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.501335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.19.65.190:37367)
0903 14:27:24.501561 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.501564 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.501575 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:24.501666 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.501668 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.501703 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.501764 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.502041 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.502491 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.503127 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.503399 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.503436 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.503478 (+ 42us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.503542 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
I20250903 14:27:24.506716 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.504515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34650)
0903 14:27:24.504634 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.504650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.504725 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.504925 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.504927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.504948 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.505079 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.505083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.505746 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.505749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.506406 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.506415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.506516 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.506530 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.506572 (+ 42us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.506623 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":384}
I20250903 14:27:24.507045 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.504592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.19.65.189:36775)
0903 14:27:24.504769 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.504772 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.504782 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:24.504812 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.504814 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.504847 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.504912 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.505154 (+ 242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.505655 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.506513 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.506859 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.506885 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.506927 (+ 42us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.506974 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"spinlock_wait_cycles":5376,"thread_start_us":74,"threads_started":1}
I20250903 14:27:24.509963 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.507850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36540)
0903 14:27:24.507959 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.507968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.508037 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.508343 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.508346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.508360 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.508561 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.508567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.509194 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.509198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.509690 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.509697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.509785 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.509798 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.509839 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.509881 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20250903 14:27:24.510432 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.507924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.19.65.188:46023)
0903 14:27:24.508157 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.508161 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.508171 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:24.508200 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.508202 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.508247 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.508335 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.508675 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.509093 (+ 418us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.509786 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.510201 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.510277 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.510310 (+ 33us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.510345 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20250903 14:27:24.517452 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.511770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.19.65.188:46023)
0903 14:27:24.511889 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.511891 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.511972 (+ 81us) server_negotiation.cc:408] Connection header received
0903 14:27:24.512067 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.512069 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.512098 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.512145 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.512942 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.513336 (+ 394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.513861 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.513957 (+ 96us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.515072 (+ 1115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.515085 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.515087 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.515106 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.516556 (+ 1450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.516802 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.516804 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.516806 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.516834 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.516951 (+ 117us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.516953 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.516954 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.517044 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.517122 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.517243 (+ 121us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.517313 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250903 14:27:24.519735 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.517991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37170)
0903 14:27:24.518202 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.518211 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.518274 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.518432 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.518434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.518444 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.518572 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.518576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.519069 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.519070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.519522 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.519528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.519598 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.519608 (+ 10us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.519637 (+ 29us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.519668 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20250903 14:27:24.520052 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.518047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.19.65.190:37367)
0903 14:27:24.518152 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.518156 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.518259 (+ 103us) server_negotiation.cc:408] Connection header received
0903 14:27:24.518338 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.518340 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.518369 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.518424 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.518636 (+ 212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.519005 (+ 369us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.519593 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.519889 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.519920 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.519953 (+ 33us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.519986 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 14:27:24.522704 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.520845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.0.0.1:34652)
0903 14:27:24.520929 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.520941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.520999 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.521154 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.521156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.521176 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.521295 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.521299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.521947 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.521950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.522409 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.522415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.522520 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.522534 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.522579 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.522620 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20250903 14:27:24.523067 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.520918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.19.65.189:36775)
0903 14:27:24.520999 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.521003 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.521014 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:24.521057 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.521060 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.521092 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.521142 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.521379 (+ 237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.521858 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.522501 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.522863 (+ 362us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.522893 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.522933 (+ 40us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.522981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20250903 14:27:24.525667 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.523841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.0.0.1:36560)
0903 14:27:24.523930 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.523939 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.523989 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.524147 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.524149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.524164 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:24.524273 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.524277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.524861 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.524863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.525383 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.525390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.525466 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.525501 (+ 35us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.525548 (+ 47us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.525599 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"spinlock_wait_cycles":384}
I20250903 14:27:24.525988 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.523882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.19.65.188:46023)
0903 14:27:24.523961 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.523964 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.523973 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:27:24.524057 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.524059 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.524086 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.524137 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:24.524364 (+ 227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.524785 (+ 421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.525497 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.525809 (+ 312us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.525840 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.525877 (+ 37us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.525927 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250903 14:27:24.526925 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26519
I20250903 14:27:24.532963 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:39959
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:37367
--webserver_port=32997
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:24.621626 27313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:24.621762 27313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:24.621812 27313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:24.624440 27313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:24.624488 27313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:24.624562 27313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:24.628144 27313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:39959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=32997
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:24.628854 27313 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:24.629489 27313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:24.634692 27321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.634688 27318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.634688 27319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:24.634846 27313 server_base.cc:1047] running on GCE node
I20250903 14:27:24.635160 27313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:24.635614 27313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:24.636765 27313 hybrid_clock.cc:648] HybridClock initialized: now 1756909644636754 us; error 22 us; skew 500 ppm
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:24.639307 27313 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20250903 14:27:24.640105 27313 webserver.cc:480] Webserver started at http://127.19.65.129:32997/ using document root <none> and password file <none>
I20250903 14:27:24.640527 27313 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:24.640578 27313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:24.643239 27313 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:24.644786 27328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.645691 27313 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:24.645764 27313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "5b407477dde04cc58b0049f4251e8073"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "1b3970025db9eb2a5e376e092534198c"
server_key_iv: "0f72c8028efcf43a7b087ed12a984a70"
server_key_version: "encryptionkey@0"
I20250903 14:27:24.646057 27313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:24.671576 27313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:24.673628 27313 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:24.673739 27313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:24.674108 27313 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:24.674815 27313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:24.674858 27313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.674892 27313 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:24.674913 27313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.683095 27313 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:39959
I20250903 14:27:24.683117 27441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:39959 every 8 connection(s)
I20250903 14:27:24.683910 27313 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:24.687556 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27313
I20250903 14:27:24.687716 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26655
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:24.696318 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:43183
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:37367
--webserver_port=39459
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 14:27:24.697834 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.686363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44721 (local address 127.19.65.189:36775)
0903 14:27:24.686481 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.686485 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.686601 (+ 116us) server_negotiation.cc:408] Connection header received
0903 14:27:24.686815 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.686818 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.686871 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.686940 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.688420 (+ 1480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689130 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.690207 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.690420 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.692564 (+ 2144us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.692578 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.692580 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.692600 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.694790 (+ 2190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.695380 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.695385 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.695388 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.695447 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.695781 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.695786 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.695791 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.695965 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.696054 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.697573 (+ 1519us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.697682 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:27:24.697937 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.685349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:58309 (local address 127.19.65.190:37367)
0903 14:27:24.685499 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.685503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.686031 (+ 528us) server_negotiation.cc:408] Connection header received
0903 14:27:24.686854 (+ 823us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.686858 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.686912 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.686964 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.688458 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.688904 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.689640 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689829 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.693584 (+ 3755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.693606 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.693613 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.693644 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.696093 (+ 2449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.696726 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.696732 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.696736 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.696785 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.697027 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.697033 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.697037 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.697217 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.697298 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.697627 (+ 329us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.697761 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 14:27:24.697937 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.686250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.19.65.129:44721)
0903 14:27:24.686526 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.686538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.686642 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.687103 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.687112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.688090 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.688330 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.688337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689238 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.689241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.689796 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.689805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689957 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.690573 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.690604 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.692402 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.694926 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.694933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.694945 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.695256 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.695574 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.695578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.695581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.695681 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.696068 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.696085 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.696329 (+ 244us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.696549 (+ 220us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.696775 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":2816,"thread_start_us":72,"threads_started":1}
I20250903 14:27:24.698299 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.686787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:41467 (local address 127.19.65.188:46023)
0903 14:27:24.686911 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.686915 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.687051 (+ 136us) server_negotiation.cc:408] Connection header received
0903 14:27:24.687155 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.687158 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.687201 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.687342 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.690616 (+ 3274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.691370 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.692198 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.692418 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.694763 (+ 2345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.694785 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.694792 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.694822 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.696867 (+ 2045us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.697247 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.697250 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.697252 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.697302 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.697561 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.697565 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.697566 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.697703 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.697762 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.698053 (+ 291us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.698157 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 14:27:24.698638 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.686572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.19.65.129:41467)
0903 14:27:24.686986 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.686997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.687080 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.689977 (+ 2897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.689984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.690173 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.690477 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.690487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.691519 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.691524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.692076 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.692085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.692543 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.693021 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.693040 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.694622 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.696976 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.696979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.696981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.697150 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.697400 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.697403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.697404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.697446 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.697822 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.697824 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.697894 (+ 70us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.698418 (+ 524us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.698514 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":67,"threads_started":1}
I20250903 14:27:24.699054 27444 heartbeater.cc:344] Connected to a master server at 127.19.65.189:36775
I20250903 14:27:24.699255 27444 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.699424 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.685572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:58309)
0903 14:27:24.685917 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.685944 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.686641 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.687103 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.687112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.687631 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.688292 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.688306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689011 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.689013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.689458 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.689476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.689957 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.690669 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.690683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.693155 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.696215 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.696221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.696225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.696617 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.696883 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.696888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.696892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.696942 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.697359 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.697364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.697438 (+ 74us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.698949 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.699074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"spinlock_wait_cycles":1280,"thread_start_us":89,"threads_started":1}
I20250903 14:27:24.699512 27442 heartbeater.cc:344] Connected to a master server at 127.19.65.188:46023
I20250903 14:27:24.699712 27442 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.699808 27444 heartbeater.cc:507] Master 127.19.65.189:36775 requested a full tablet report, sending...
I20250903 14:27:24.699874 27442 heartbeater.cc:507] Master 127.19.65.188:46023 requested a full tablet report, sending...
I20250903 14:27:24.700500 27443 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:24.700579 27443 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.700726 27443 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
I20250903 14:27:24.701432 27249 ts_manager.cc:194] Registered new tserver with Master: 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:24.701515 26455 ts_manager.cc:194] Re-registered known tserver with Master: 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:24.702239 27061 ts_manager.cc:194] Registered new tserver with Master: 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:24.702286 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:58309
W20250903 14:27:24.789223 27449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:24.789366 27449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:24.789448 27449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:24.792148 27449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:24.792196 27449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:24.792291 27449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:24.795979 27449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:43183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=39459
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:24.796720 27449 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:24.797317 27449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:24.802350 27455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.802341 27454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.802330 27457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:24.802595 27449 server_base.cc:1047] running on GCE node
I20250903 14:27:24.802866 27449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:24.803261 27449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:24.804430 27449 hybrid_clock.cc:648] HybridClock initialized: now 1756909644804410 us; error 26 us; skew 500 ppm
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:24.806735 27449 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20250903 14:27:24.807492 27449 webserver.cc:480] Webserver started at http://127.19.65.130:39459/ using document root <none> and password file <none>
I20250903 14:27:24.807950 27449 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:24.808000 27449 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:24.810860 27449 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:27:24.812390 27464 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.813308 27449 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:24.813403 27449 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "d99be9c94fbc410ebf5f1c2f503e59b0"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "b2c2066b6bf8b8e316162dd23231026a"
server_key_iv: "dc894932b46d9877f2157a61db3d2da7"
server_key_version: "encryptionkey@0"
I20250903 14:27:24.813737 27449 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:24.838079 27449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:24.840010 27449 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:24.840145 27449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:24.840552 27449 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:24.841184 27449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:24.841223 27449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.841256 27449 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:24.841285 27449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.848892 27449 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43183
I20250903 14:27:24.848929 27577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:43183 every 8 connection(s)
I20250903 14:27:24.850100 27449 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:24.851676 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27449
I20250903 14:27:24.851855 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26791
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:24.860661 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:38077
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:37367
--webserver_port=41003
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023 with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20250903 14:27:24.863981 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.851560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:42095 (local address 127.19.65.190:37367)
0903 14:27:24.851675 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.851680 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.852640 (+ 960us) server_negotiation.cc:408] Connection header received
0903 14:27:24.853631 (+ 991us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.853637 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.853699 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.853751 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.855479 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.856189 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.856975 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.857146 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.859746 (+ 2600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.859763 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.859766 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.859788 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.861176 (+ 1388us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.861682 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.861685 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.861686 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.861724 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.862646 (+ 922us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.862870 (+ 224us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.862877 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.863050 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.863131 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.863668 (+ 537us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.863783 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:27:24.864825 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.853155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52369 (local address 127.19.65.188:46023)
0903 14:27:24.853304 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.853310 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.853693 (+ 383us) server_negotiation.cc:408] Connection header received
0903 14:27:24.853793 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.853799 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.853857 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.853942 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.855448 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.856231 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.857999 (+ 1768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.858221 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.861554 (+ 3333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.861574 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.861578 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.861605 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.863318 (+ 1713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863724 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.863727 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.863729 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.863769 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863997 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.864000 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.864001 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.864155 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.864212 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.864544 (+ 332us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.864653 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 14:27:24.865085 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.853290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33115 (local address 127.19.65.189:36775)
0903 14:27:24.853396 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:24.853400 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:24.854046 (+ 646us) server_negotiation.cc:408] Connection header received
0903 14:27:24.855494 (+ 1448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:24.855514 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:24.855580 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:24.855627 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:24.856283 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.856960 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.858847 (+ 1887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.859051 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.861958 (+ 2907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:24.861978 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:24.861982 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:24.862013 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:24.863595 (+ 1582us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863968 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.863973 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.863977 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.864017 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:24.864243 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:24.864245 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:24.864247 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:24.864440 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:24.864511 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:24.864823 (+ 312us) server_negotiation.cc:300] Negotiation successful
0903 14:27:24.864940 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 14:27:24.865871 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.851795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:42095)
0903 14:27:24.852526 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.852562 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.853394 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.854018 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.854029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.854551 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.855308 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.855322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.856313 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.856316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.856859 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.856866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.857308 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.857879 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.857924 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.859630 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.861276 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.861280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.861286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.861590 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.862015 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.862018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.862020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.862110 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.863206 (+ 1096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.863213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.863416 (+ 203us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.864312 (+ 896us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.864603 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"spinlock_wait_cycles":1920,"thread_start_us":103,"threads_started":1}
I20250903 14:27:24.865871 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.852821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.19.65.130:33115)
0903 14:27:24.853980 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.853991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.854396 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.855734 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.855736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.855895 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.856143 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.856149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.858337 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.858341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.858746 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.858754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.858849 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.859773 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.859787 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.861830 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.863700 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.863709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.863877 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.864111 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.864113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.864115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.864154 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.864578 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.864580 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.864652 (+ 72us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.865257 (+ 605us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.865364 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":63,"threads_started":1}
I20250903 14:27:24.865871 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:24.852535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.19.65.130:52369)
0903 14:27:24.853628 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:24.853640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:24.853715 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:24.854104 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:24.854107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:24.854553 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:24.855308 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.855322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.857052 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:24.857055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:24.857869 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:24.857878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:24.857995 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:24.859384 (+ 1389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:24.859402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:24.861169 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:24.863437 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.863443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.863632 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.863865 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:24.863867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:24.863869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:24.863915 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:24.864267 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:24.864270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:24.864334 (+ 64us) client_negotiation.cc:769] Sending connection context
0903 14:27:24.865003 (+ 669us) client_negotiation.cc:240] Negotiation successful
0903 14:27:24.865134 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"spinlock_wait_cycles":30848,"thread_start_us":70,"threads_started":1}
I20250903 14:27:24.866809 27580 heartbeater.cc:344] Connected to a master server at 127.19.65.189:36775
I20250903 14:27:24.866883 27579 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:24.867123 27578 heartbeater.cc:344] Connected to a master server at 127.19.65.188:46023
I20250903 14:27:24.867089 27579 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.867209 27578 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.867089 27580 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:24.867599 27579 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
I20250903 14:27:24.867597 27578 heartbeater.cc:507] Master 127.19.65.188:46023 requested a full tablet report, sending...
I20250903 14:27:24.867762 27580 heartbeater.cc:507] Master 127.19.65.189:36775 requested a full tablet report, sending...
I20250903 14:27:24.868575 27061 ts_manager.cc:194] Registered new tserver with Master: d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183)
I20250903 14:27:24.868575 27249 ts_manager.cc:194] Registered new tserver with Master: d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183)
I20250903 14:27:24.868661 26455 ts_manager.cc:194] Re-registered known tserver with Master: d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183)
I20250903 14:27:24.869374 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:42095
W20250903 14:27:24.952001 27585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:24.952163 27585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:24.952215 27585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:24.954982 27585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:24.955031 27585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:24.955103 27585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:24.958477 27585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:38077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=41003
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:37367,127.19.65.189:36775,127.19.65.188:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:24.959146 27585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:24.959764 27585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:24.964628 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.964663 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:24.964610 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:24.964818 27585 server_base.cc:1047] running on GCE node
I20250903 14:27:24.965191 27585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:24.965682 27585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:24.966818 27585 hybrid_clock.cc:648] HybridClock initialized: now 1756909644966811 us; error 23 us; skew 500 ppm
Sep 03 14:27:24 dist-test-slave-4wpb krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:24.969019 27585 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20250903 14:27:24.969825 27585 webserver.cc:480] Webserver started at http://127.19.65.131:41003/ using document root <none> and password file <none>
I20250903 14:27:24.970227 27585 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:24.970275 27585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:24.972892 27585 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:24.974509 27601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.975369 27585 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:24.975450 27585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "f8fea8909934484380f011507ceb0a66"
format_stamp: "Formatted at 2025-09-03 14:27:22 on dist-test-slave-4wpb"
server_key: "d3c0f646bb749087ee3ea9ee057b87c8"
server_key_iv: "3940d483f2d4890de3e8334c2499fd9e"
server_key_version: "encryptionkey@0"
I20250903 14:27:24.975780 27585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:24.989512 27585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:24.991457 27585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:24.991588 27585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:24.991976 27585 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:24.992650 27585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:24.992690 27585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:24.992725 27585 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:24.992751 27585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:25.000126 27585 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38077
I20250903 14:27:25.000161 27714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:38077 every 8 connection(s)
I20250903 14:27:25.000892 27585 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:25.005492 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27585
Sep 03 14:27:25 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
Sep 03 14:27:25 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Sep 03 14:27:25 dist-test-slave-4wpb krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909644, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20250903 14:27:25.015475 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.002388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:57859)
0903 14:27:25.002724 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.002753 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.003467 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.003918 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.003927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.004776 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:25.005019 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.005028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.006993 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.006998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.007459 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.007467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.007553 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.007964 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:25.007980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:25.009422 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:25.011912 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.011916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.011924 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.012186 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.013234 (+ 1048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.013237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.013238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.013324 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.014006 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:25.014012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:25.014268 (+ 256us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.014516 (+ 248us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.014752 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"spinlock_wait_cycles":1920,"thread_start_us":92,"threads_started":1}
I20250903 14:27:25.015872 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.002222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57859 (local address 127.19.65.190:37367)
0903 14:27:25.002337 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.002340 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.002829 (+ 489us) server_negotiation.cc:408] Connection header received
0903 14:27:25.003649 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.003652 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.003716 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.003782 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.005121 (+ 1339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.005861 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.007556 (+ 1695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.007767 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.009673 (+ 1906us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.009689 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.009692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.009719 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.011802 (+ 2083us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.012679 (+ 877us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.012684 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.012687 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.012993 (+ 306us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.013422 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.013427 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.013430 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.013918 (+ 488us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.014006 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.015596 (+ 1590us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.015741 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:27:25.015479 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.003025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46023 (local address 127.19.65.131:38533)
0903 14:27:25.003236 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.003252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.003710 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.003930 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.003932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.004403 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:25.004978 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.004986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.005978 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.005981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.006516 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.006524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.007225 (+ 701us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.007928 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:25.007945 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:25.009806 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:25.013039 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.013042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.013058 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.013312 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.014204 (+ 892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.014209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.014213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.014268 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.014748 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:25.014752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:25.014843 (+ 91us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.014953 (+ 110us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.015095 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"spinlock_wait_cycles":7168,"thread_start_us":73,"threads_started":1}
I20250903 14:27:25.016463 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.003139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38533 (local address 127.19.65.188:46023)
0903 14:27:25.003281 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.003286 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.003300 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:25.003803 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.003809 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.003857 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.003924 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.005125 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.005802 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.006637 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.007109 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.009926 (+ 2817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.009943 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.009947 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.009975 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.012950 (+ 2975us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.014048 (+ 1098us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.014051 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.014053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.014105 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.014392 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.014397 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.014400 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.014602 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.014699 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.016210 (+ 1511us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.016322 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:27:25.016760 27716 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37367
I20250903 14:27:25.016968 27716 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:25.017009 27715 heartbeater.cc:344] Connected to a master server at 127.19.65.188:46023
I20250903 14:27:25.017082 27715 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:25.017503 27715 heartbeater.cc:507] Master 127.19.65.188:46023 requested a full tablet report, sending...
I20250903 14:27:25.017529 27716 heartbeater.cc:507] Master 127.19.65.190:37367 requested a full tablet report, sending...
I20250903 14:27:25.018520 26455 ts_manager.cc:194] Re-registered known tserver with Master: f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:25.019130 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.008288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.19.65.190:37367)
0903 14:27:25.009827 (+ 1539us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.009832 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.009846 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:25.009878 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.009883 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.009922 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.010954 (+ 1032us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.011050 (+ 96us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.011750 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.012777 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.014013 (+ 1236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.014093 (+ 80us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.014107 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.014110 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.014127 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.016969 (+ 2842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.017842 (+ 873us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.017845 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.017847 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.017888 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.018150 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.018153 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.018154 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.018310 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.018845 (+ 535us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.018905 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.019003 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472,"thread_start_us":64,"threads_started":1}
I20250903 14:27:25.019424 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.003337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:36775 (local address 127.19.65.131:36495)
0903 14:27:25.003533 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.003546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.003634 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.003936 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.003939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.004409 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:25.004919 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.004931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.005954 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.005957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.006390 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.006397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.006589 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.007074 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:25.007111 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:25.009789 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:25.013153 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.013158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.013160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.018171 (+ 5011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.018419 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:25.018421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:25.018423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:25.018477 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:25.018848 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:25.018851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:25.018915 (+ 64us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.019186 (+ 271us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.019293 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20250903 14:27:25.019577 27249 ts_manager.cc:194] Registered new tserver with Master: f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:25.019779 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.003476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:36495 (local address 127.19.65.189:36775)
0903 14:27:25.003599 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.003604 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.003617 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:25.003721 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.003724 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.003772 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.003835 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.005053 (+ 1218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.005825 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.006530 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.006691 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.009906 (+ 3215us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.009921 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.009925 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.009949 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.012569 (+ 2620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.018275 (+ 5706us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.018278 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.018279 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.018318 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.018569 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.018572 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.018574 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.018718 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.018788 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.019081 (+ 293us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.019649 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250903 14:27:25.020258 27717 heartbeater.cc:344] Connected to a master server at 127.19.65.189:36775
I20250903 14:27:25.020334 27717 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:25.019148 26455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:57859
I20250903 14:27:25.020498 27717 heartbeater.cc:507] Master 127.19.65.189:36775 requested a full tablet report, sending...
I20250903 14:27:25.021808 27061 ts_manager.cc:194] Registered new tserver with Master: f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:25.022918 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.009545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.19.65.188:46023)
0903 14:27:25.011028 (+ 1483us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.011033 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.011045 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:25.011073 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.011077 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.011116 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.011920 (+ 804us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.012003 (+ 83us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.012689 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.013672 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.014717 (+ 1045us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.014807 (+ 90us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.014821 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.014824 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.014840 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.019496 (+ 4656us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.021608 (+ 2112us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.021611 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.021614 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.021658 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.021884 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.021887 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.021888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.022036 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.022624 (+ 588us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.022684 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.022794 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1415,"thread_start_us":62,"threads_started":1}
I20250903 14:27:25.022341 26455 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:27:25.024432 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.008581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.19.65.189:36775)
0903 14:27:25.008827 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.008831 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.010797 (+ 1966us) server_negotiation.cc:408] Connection header received
0903 14:27:25.010928 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.010931 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.010997 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.013433 (+ 2436us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.013539 (+ 106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.015319 (+ 1780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.018624 (+ 3305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.020148 (+ 1524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.020225 (+ 77us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.021016 (+ 791us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.021023 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.021041 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.022905 (+ 1864us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.023402 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.023407 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.023410 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.023448 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.023687 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.023689 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.023690 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.023798 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.023881 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.024060 (+ 179us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.024137 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":62,"threads_started":1}
W20250903 14:27:25.025094 26455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:25.038260 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.034509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:43183 (local address 127.0.0.1:33786)
0903 14:27:25.034730 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.034746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.034829 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.035830 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.035832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.035846 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:25.036069 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.036077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037189 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.037192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.037965 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.037972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.038056 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.038068 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.038106 (+ 38us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.038155 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20250903 14:27:25.038520 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.035046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:38077 (local address 127.0.0.1:33920)
0903 14:27:25.035267 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.035282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.035378 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.036387 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.036389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.036405 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:25.036597 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.036603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037535 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.037538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.038054 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.038062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.038164 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.038178 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.038219 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.038269 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20250903 14:27:25.038868 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.034638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.19.65.130:43183)
0903 14:27:25.034978 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.034985 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.035009 (+ 24us) server_negotiation.cc:408] Connection header received
0903 14:27:25.035093 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.035101 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.035283 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.035851 (+ 568us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:25.036179 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037094 (+ 915us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.038081 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.038519 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.038618 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.038678 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.038753 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
I20250903 14:27:25.039345 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.035153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.19.65.131:38077)
0903 14:27:25.035494 (+ 341us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.035500 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.035520 (+ 20us) server_negotiation.cc:408] Connection header received
0903 14:27:25.035582 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.035588 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.035729 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.035835 (+ 106us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:25.036701 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037423 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.038673 (+ 1250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.039074 (+ 401us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.039151 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.039196 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.039251 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20250903 14:27:25.039757 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.035367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:39959 (local address 127.0.0.1:53520)
0903 14:27:25.035569 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.035581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.035653 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.036633 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.036635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.036651 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:25.036822 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.036828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037893 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.037896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.038563 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.038571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.039540 (+ 969us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.039551 (+ 11us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.039613 (+ 62us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.039662 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20250903 14:27:25.039971 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.035509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.19.65.129:39959)
0903 14:27:25.036131 (+ 622us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.036137 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.036156 (+ 19us) server_negotiation.cc:408] Connection header received
0903 14:27:25.036229 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.036233 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.036372 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.036497 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:25.036920 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.037787 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.038877 (+ 1090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.039399 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.039490 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.039662 (+ 172us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.039730 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":107,"threads_started":1}
I20250903 14:27:25.040766 27512 tablet_service.cc:1468] Processing CreateTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 (DEFAULT_TABLE table=test-table [id=2842e911fbfe466c90321b7689c372a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:25.040997 27649 tablet_service.cc:1468] Processing CreateTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 (DEFAULT_TABLE table=test-table [id=2842e911fbfe466c90321b7689c372a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:25.041453 27512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0c50b07104e4997b69f73ac5990a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:25.041632 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0c50b07104e4997b69f73ac5990a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:25.046731 27747 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Bootstrap starting.
I20250903 14:27:25.047452 27376 tablet_service.cc:1468] Processing CreateTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 (DEFAULT_TABLE table=test-table [id=2842e911fbfe466c90321b7689c372a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:25.048024 27376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0c50b07104e4997b69f73ac5990a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:25.048507 27747 tablet_bootstrap.cc:654] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:25.049059 27747 log.cc:826] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:25.050472 27747 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: No bootstrap required, opened a new log
I20250903 14:27:25.050631 27747 ts_tablet_manager.cc:1397] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:25.052044 27749 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Bootstrap starting.
I20250903 14:27:25.052496 27746 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Bootstrap starting.
I20250903 14:27:25.052585 27747 raft_consensus.cc:357] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.052744 27747 raft_consensus.cc:383] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:25.052796 27747 raft_consensus.cc:738] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99be9c94fbc410ebf5f1c2f503e59b0, State: Initialized, Role: FOLLOWER
I20250903 14:27:25.053144 27747 consensus_queue.cc:260] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.053828 27747 ts_tablet_manager.cc:1428] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250903 14:27:25.054006 27579 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:25.054489 27746 tablet_bootstrap.cc:654] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:25.054522 27749 tablet_bootstrap.cc:654] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:25.055261 27746 log.cc:826] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:25.055261 27749 log.cc:826] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:25.056696 27746 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: No bootstrap required, opened a new log
I20250903 14:27:25.056696 27749 tablet_bootstrap.cc:492] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: No bootstrap required, opened a new log
I20250903 14:27:25.056833 27749 ts_tablet_manager.cc:1397] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:27:25.056833 27746 ts_tablet_manager.cc:1397] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250903 14:27:25.059178 27746 raft_consensus.cc:357] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.059187 27749 raft_consensus.cc:357] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.059346 27746 raft_consensus.cc:383] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:25.059346 27749 raft_consensus.cc:383] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:25.059396 27749 raft_consensus.cc:738] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b407477dde04cc58b0049f4251e8073, State: Initialized, Role: FOLLOWER
I20250903 14:27:25.059396 27746 raft_consensus.cc:738] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fea8909934484380f011507ceb0a66, State: Initialized, Role: FOLLOWER
I20250903 14:27:25.059752 27746 consensus_queue.cc:260] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.059752 27749 consensus_queue.cc:260] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.060269 27443 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:25.060407 27746 ts_tablet_manager.cc:1428] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:27:25.060907 27716 heartbeater.cc:499] Master 127.19.65.190:37367 was elected leader, sending a full tablet report...
I20250903 14:27:25.062520 27749 ts_tablet_manager.cc:1428] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250903 14:27:25.101569 27581 tablet.cc:2378] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 14:27:25.185149 27445 tablet.cc:2378] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:25.225724 27750 raft_consensus.cc:491] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:25.225858 27750 raft_consensus.cc:513] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.226763 27750 leader_election.cc:290] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959), f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:25.229967 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.226954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:39959 (local address 127.19.65.130:60663)
0903 14:27:25.227156 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.227170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.227260 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.227505 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.227508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.227536 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:25.227826 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.227832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.228707 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.228710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.229579 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.229588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.229698 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.229720 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.229770 (+ 50us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.229824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"spinlock_wait_cycles":9344}
I20250903 14:27:25.229967 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.226954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:38077 (local address 127.19.65.130:46331)
0903 14:27:25.227204 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:25.227215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:25.227266 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:25.227505 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:25.227508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:25.227535 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:25.227826 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.227832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.228707 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.228710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:25.229579 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:25.229588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.229708 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.229720 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 14:27:25.229770 (+ 50us) client_negotiation.cc:240] Negotiation successful
0903 14:27:25.229824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":37,"spinlock_wait_cycles":66176}
I20250903 14:27:25.230329 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.227031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60663 (local address 127.19.65.129:39959)
0903 14:27:25.227161 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.227169 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.227212 (+ 43us) server_negotiation.cc:408] Connection header received
0903 14:27:25.227361 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.227364 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.227411 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.227483 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:25.227933 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.228611 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.229692 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.230121 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.230151 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.230193 (+ 42us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.230237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 14:27:25.230329 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.227083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:46331 (local address 127.19.65.131:38077)
0903 14:27:25.227220 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.227226 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.227242 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:27:25.227355 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.227358 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.227404 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.227479 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:25.227933 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.228611 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.229692 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.230121 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.230161 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.230197 (+ 36us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.230238 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:27:25.230779 27396 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66c0c50b07104e4997b69f73ac5990a1" candidate_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b407477dde04cc58b0049f4251e8073" is_pre_election: true
I20250903 14:27:25.230791 27669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66c0c50b07104e4997b69f73ac5990a1" candidate_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8fea8909934484380f011507ceb0a66" is_pre_election: true
I20250903 14:27:25.230974 27396 raft_consensus.cc:2466] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99be9c94fbc410ebf5f1c2f503e59b0 in term 0.
I20250903 14:27:25.230990 27669 raft_consensus.cc:2466] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99be9c94fbc410ebf5f1c2f503e59b0 in term 0.
I20250903 14:27:25.231326 27466 leader_election.cc:304] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b407477dde04cc58b0049f4251e8073, d99be9c94fbc410ebf5f1c2f503e59b0; no voters:
I20250903 14:27:25.231499 27750 raft_consensus.cc:2802] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:25.231562 27750 raft_consensus.cc:491] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:25.231588 27750 raft_consensus.cc:3058] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:25.232419 27750 raft_consensus.cc:513] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.232697 27750 leader_election.cc:290] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959), f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077)
I20250903 14:27:25.232981 27396 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66c0c50b07104e4997b69f73ac5990a1" candidate_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b407477dde04cc58b0049f4251e8073"
I20250903 14:27:25.232985 27669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66c0c50b07104e4997b69f73ac5990a1" candidate_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8fea8909934484380f011507ceb0a66"
I20250903 14:27:25.233069 27396 raft_consensus.cc:3058] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:25.233103 27669 raft_consensus.cc:3058] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:25.233868 27396 raft_consensus.cc:2466] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99be9c94fbc410ebf5f1c2f503e59b0 in term 1.
I20250903 14:27:25.234019 27669 raft_consensus.cc:2466] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99be9c94fbc410ebf5f1c2f503e59b0 in term 1.
I20250903 14:27:25.234267 27466 leader_election.cc:304] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b407477dde04cc58b0049f4251e8073, d99be9c94fbc410ebf5f1c2f503e59b0; no voters:
I20250903 14:27:25.234443 27750 raft_consensus.cc:2802] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:25.234611 27750 raft_consensus.cc:695] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 LEADER]: Becoming Leader. State: Replica: d99be9c94fbc410ebf5f1c2f503e59b0, State: Running, Role: LEADER
I20250903 14:27:25.234885 27750 consensus_queue.cc:237] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } }
I20250903 14:27:25.237843 26453 catalog_manager.cc:5582] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } health_report { overall_health: UNKNOWN } } }
W20250903 14:27:25.251799 27718 tablet.cc:2378] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:25.269126 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.266706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.19.65.130:43183)
0903 14:27:25.266814 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.266818 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.266831 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:25.266902 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.266904 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.266944 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.267017 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:25.267300 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.267697 (+ 397us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.268238 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.268401 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.268474 (+ 73us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:25.268816 (+ 342us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:25.268913 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.268982 (+ 69us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.269023 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 14:27:25.273415 27669 raft_consensus.cc:1273] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Refusing update from remote peer d99be9c94fbc410ebf5f1c2f503e59b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:25.273416 27396 raft_consensus.cc:1273] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Refusing update from remote peer d99be9c94fbc410ebf5f1c2f503e59b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:25.273960 27750 consensus_queue.cc:1035] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:25.274137 27755 consensus_queue.cc:1035] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:25.280676 27758 mvcc.cc:204] Tried to move back new op lower bound from 7196301907034882048 to 7196301906884227072. Current Snapshot: MvccSnapshot[applied={T|T < 7196301907034882048}]
I20250903 14:27:25.282718 27760 mvcc.cc:204] Tried to move back new op lower bound from 7196301907034882048 to 7196301906884227072. Current Snapshot: MvccSnapshot[applied={T|T < 7196301907034882048}]
I20250903 14:27:25.290623 26453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:25.290729 26453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:25.294576 26453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 55f880c1f5e3421ab6ec375c6d4932d1: Sending DeleteTablet for 3 replicas of tablet 66c0c50b07104e4997b69f73ac5990a1
I20250903 14:27:25.295233 27376 tablet_service.cc:1515] Processing DeleteTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:25 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:53520
I20250903 14:27:25.295666 27767 tablet_replica.cc:331] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: stopping tablet replica
I20250903 14:27:25.295855 27512 tablet_service.cc:1515] Processing DeleteTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:25 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33786
I20250903 14:27:25.295900 27767 raft_consensus.cc:2241] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:25.296177 27767 raft_consensus.cc:2270] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:25.296289 27768 tablet_replica.cc:331] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: stopping tablet replica
I20250903 14:27:25.296526 27649 tablet_service.cc:1515] Processing DeleteTablet for tablet 66c0c50b07104e4997b69f73ac5990a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:25 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33920
I20250903 14:27:25.296904 27768 raft_consensus.cc:2241] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:25.297252 27769 tablet_replica.cc:331] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: stopping tablet replica
I20250903 14:27:25.297272 27768 raft_consensus.cc:2270] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:25.297456 27769 raft_consensus.cc:2241] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:25.297596 27767 ts_tablet_manager.cc:1905] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:25.297822 27769 raft_consensus.cc:2270] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:25.298803 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26423
I20250903 14:27:25.299358 27769 ts_tablet_manager.cc:1905] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:25.299651 27768 ts_tablet_manager.cc:1905] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:25.300212 27767 ts_tablet_manager.cc:1918] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:25.300292 27767 log.cc:1199] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/66c0c50b07104e4997b69f73ac5990a1
I20250903 14:27:25.300587 27767 ts_tablet_manager.cc:1939] T 66c0c50b07104e4997b69f73ac5990a1 P 5b407477dde04cc58b0049f4251e8073: Deleting consensus metadata
I20250903 14:27:25.301676 27769 ts_tablet_manager.cc:1918] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:25.301743 27769 log.cc:1199] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/66c0c50b07104e4997b69f73ac5990a1
I20250903 14:27:25.302022 27769 ts_tablet_manager.cc:1939] T 66c0c50b07104e4997b69f73ac5990a1 P f8fea8909934484380f011507ceb0a66: Deleting consensus metadata
I20250903 14:27:25.302430 27768 ts_tablet_manager.cc:1918] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:25.302498 27768 log.cc:1199] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/66c0c50b07104e4997b69f73ac5990a1
I20250903 14:27:25.302779 27768 ts_tablet_manager.cc:1939] T 66c0c50b07104e4997b69f73ac5990a1 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting consensus metadata
W20250903 14:27:25.307789 27329 connection.cc:537] server connection from 127.0.0.1:53520 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:27:25.307827 27604 connection.cc:537] server connection from 127.0.0.1:33920 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:27:25.308140 27465 connection.cc:537] server connection from 127.0.0.1:33786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:27:25.318331 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.310507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.19.65.189:36775)
0903 14:27:25.310608 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.310612 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.310625 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:25.310694 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.310697 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.310756 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.310806 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.311908 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.312787 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.313556 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.313766 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.315006 (+ 1240us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.315024 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.315031 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.315062 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.316703 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.317041 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.317044 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.317045 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.317080 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.317609 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.317611 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.317612 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.317746 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.317861 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.318078 (+ 217us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.318178 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 14:27:25.318431 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:25.310752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.19.65.188:46023)
0903 14:27:25.310861 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:25.310865 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:25.310970 (+ 105us) server_negotiation.cc:408] Connection header received
0903 14:27:25.311053 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:25.311056 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:25.311098 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:25.311186 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:25.311981 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.312471 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:25.313162 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:25.313939 (+ 777us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:25.315325 (+ 1386us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:25.315340 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:25.315344 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:25.315369 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:25.317062 (+ 1693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.317390 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.317393 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.317395 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.317438 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:25.317749 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:25.317752 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:25.317753 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:25.317906 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:25.318003 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:25.318192 (+ 189us) server_negotiation.cc:300] Negotiation successful
0903 14:27:25.318289 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:27:26.285279 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.284782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:53163)
0903 14:27:26.285086 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.285161 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20250903 14:27:26.285840 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.285357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:41709)
0903 14:27:26.285673 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.285742 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":63,"threads_started":1}
W20250903 14:27:26.286057 27716 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:37367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
W20250903 14:27:26.286260 27579 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:37367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.286633 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.286187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:34485)
0903 14:27:26.286485 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.286552 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":64,"threads_started":1}
W20250903 14:27:26.286845 27443 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:37367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.795603 27779 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55f880c1f5e3421ab6ec375c6d4932d1)
I20250903 14:27:26.795737 27780 raft_consensus.cc:491] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55f880c1f5e3421ab6ec375c6d4932d1)
I20250903 14:27:26.795818 27779 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:26.795893 27780 raft_consensus.cc:513] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:26.796690 27779 leader_election.cc:290] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189:36775)
I20250903 14:27:26.796690 27780 leader_election.cc:290] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023)
I20250903 14:27:26.797159 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.796791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37590)
0903 14:27:26.797004 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.797069 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20250903 14:27:26.797189 27071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f0c7257d43944b6b75d66a871ac979e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" is_pre_election: true
I20250903 14:27:26.797289 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.796794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37588)
0903 14:27:26.797112 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.797161 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":44,"threads_started":1}
I20250903 14:27:26.797356 27071 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f0c7257d43944b6b75d66a871ac979e in term 1.
I20250903 14:27:26.797189 27259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f0c7257d43944b6b75d66a871ac979e" is_pre_election: true
W20250903 14:27:26.797400 27046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.797492 27259 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a76ae20d7c904e3ea2ebe7e94df5dfae in term 1.
W20250903 14:27:26.797675 27234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.797814 27046 leader_election.cc:304] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0c7257d43944b6b75d66a871ac979e, a76ae20d7c904e3ea2ebe7e94df5dfae; no voters:
I20250903 14:27:26.797962 27234 leader_election.cc:304] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0c7257d43944b6b75d66a871ac979e, a76ae20d7c904e3ea2ebe7e94df5dfae; no voters:
I20250903 14:27:26.797986 27780 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 14:27:26.798029 27780 raft_consensus.cc:491] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55f880c1f5e3421ab6ec375c6d4932d1)
I20250903 14:27:26.798092 27780 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:27:26.798338 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.798073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37604)
0903 14:27:26.798238 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.798272 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20250903 14:27:26.798390 27779 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250903 14:27:26.798441 27046 leader_election.cc:336] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.798446 27779 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55f880c1f5e3421ab6ec375c6d4932d1)
I20250903 14:27:26.798497 27779 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:27:26.798508 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.798269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37606)
0903 14:27:26.798386 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.798427 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20250903 14:27:26.798581 27234 leader_election.cc:336] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.799257 27780 raft_consensus.cc:513] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:26.799571 27780 leader_election.cc:290] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 election: Requested vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023)
I20250903 14:27:26.799723 27779 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:26.799921 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.799645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37614)
0903 14:27:26.799826 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.799867 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20250903 14:27:26.799927 27259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f0c7257d43944b6b75d66a871ac979e"
I20250903 14:27:26.800057 27259 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a76ae20d7c904e3ea2ebe7e94df5dfae in current term 2: Already voted for candidate 9f0c7257d43944b6b75d66a871ac979e in this term.
I20250903 14:27:26.800276 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.800090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37616)
0903 14:27:26.800181 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.800222 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20250903 14:27:26.800438 27071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f0c7257d43944b6b75d66a871ac979e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae"
I20250903 14:27:26.800318 27779 leader_election.cc:290] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 election: Requested vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), a76ae20d7c904e3ea2ebe7e94df5dfae (127.19.65.189:36775)
I20250903 14:27:26.800518 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.800249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37628)
0903 14:27:26.800428 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.800469 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20250903 14:27:26.800551 27071 raft_consensus.cc:2391] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f0c7257d43944b6b75d66a871ac979e in current term 2: Already voted for candidate a76ae20d7c904e3ea2ebe7e94df5dfae in this term.
W20250903 14:27:26.800588 27046 leader_election.cc:336] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.800640 27046 leader_election.cc:304] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a76ae20d7c904e3ea2ebe7e94df5dfae; no voters: 55f880c1f5e3421ab6ec375c6d4932d1, 9f0c7257d43944b6b75d66a871ac979e
I20250903 14:27:26.800760 27780 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 14:27:26.801035 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:26.800759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37642)
0903 14:27:26.800919 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:26.800958 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":51,"threads_started":1}
W20250903 14:27:26.801155 27234 leader_election.cc:336] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:26.801244 27234 leader_election.cc:304] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f0c7257d43944b6b75d66a871ac979e; no voters: 55f880c1f5e3421ab6ec375c6d4932d1, a76ae20d7c904e3ea2ebe7e94df5dfae
I20250903 14:27:26.801430 27779 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 14:27:27.287232 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:27.286729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:39977)
0903 14:27:27.287062 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:27.287117 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20250903 14:27:27.287338 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:27.286553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:47591)
0903 14:27:27.287176 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:27.287231 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":99,"threads_started":1}
I20250903 14:27:27.287940 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:27.287379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:45299)
0903 14:27:27.287802 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:27.287856 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":62,"threads_started":1}
I20250903 14:27:28.288529 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.287999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:38899)
0903 14:27:28.288364 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.288421 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20250903 14:27:28.288877 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.288032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:38735)
0903 14:27:28.288737 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.288793 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":98,"threads_started":1}
I20250903 14:27:28.289004 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.288463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:45907)
0903 14:27:28.288862 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.288910 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":58,"threads_started":1}
I20250903 14:27:28.341624 27792 raft_consensus.cc:491] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:28.341712 27792 raft_consensus.cc:513] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:28.342031 27792 leader_election.cc:290] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023)
I20250903 14:27:28.342440 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.342136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37860)
0903 14:27:28.342303 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.342355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":49,"threads_started":1}
I20250903 14:27:28.342499 27259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f0c7257d43944b6b75d66a871ac979e" is_pre_election: true
I20250903 14:27:28.342625 27259 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a76ae20d7c904e3ea2ebe7e94df5dfae in term 2.
I20250903 14:27:28.343005 27046 leader_election.cc:304] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0c7257d43944b6b75d66a871ac979e, a76ae20d7c904e3ea2ebe7e94df5dfae; no voters:
I20250903 14:27:28.343101 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.342899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37868)
0903 14:27:28.342987 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.343027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20250903 14:27:28.343173 27046 leader_election.cc:336] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:28.343202 27792 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 14:27:28.343269 27792 raft_consensus.cc:491] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:28.343315 27792 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 2 FOLLOWER]: Advancing to term 3
I20250903 14:27:28.345206 27792 raft_consensus.cc:513] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:28.345525 27792 leader_election.cc:290] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 election: Requested vote from peers 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367), 9f0c7257d43944b6b75d66a871ac979e (127.19.65.188:46023)
I20250903 14:27:28.345840 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.345623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37876)
0903 14:27:28.345710 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.345759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20250903 14:27:28.346019 27259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f0c7257d43944b6b75d66a871ac979e"
I20250903 14:27:28.346138 27259 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 2 FOLLOWER]: Advancing to term 3
I20250903 14:27:28.346318 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.346166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37880)
0903 14:27:28.346238 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.346279 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20250903 14:27:28.346400 27046 leader_election.cc:336] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
I20250903 14:27:28.346901 27259 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a76ae20d7c904e3ea2ebe7e94df5dfae in term 3.
I20250903 14:27:28.347134 27046 leader_election.cc:304] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f0c7257d43944b6b75d66a871ac979e, a76ae20d7c904e3ea2ebe7e94df5dfae; no voters: 55f880c1f5e3421ab6ec375c6d4932d1
I20250903 14:27:28.347294 27792 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 3 FOLLOWER]: Leader election won for term 3
I20250903 14:27:28.347431 27792 raft_consensus.cc:695] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [term 3 LEADER]: Becoming Leader. State: Replica: a76ae20d7c904e3ea2ebe7e94df5dfae, State: Running, Role: LEADER
I20250903 14:27:28.347638 27792 consensus_queue.cc:237] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } }
I20250903 14:27:28.348484 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a76ae20d7c904e3ea2ebe7e94df5dfae. Latest consensus state: current_term: 3 leader_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:28.348577 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:28.348807 27797 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:28.350672 27797 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:28.350992 27797 catalog_manager.cc:1261] Loaded cluster ID: 8be42f9d5dde470aa2cf5c0895508058
I20250903 14:27:28.351042 27797 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:28.351501 27797 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:28.351852 27797 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Loaded TSK: 0
I20250903 14:27:28.352502 27797 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:27:28.415905 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:28.418054 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:28.420295 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.420099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37890)
0903 14:27:28.420179 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.420226 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20250903 14:27:28.420472 27259 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [term 3 FOLLOWER]: Refusing update from remote peer a76ae20d7c904e3ea2ebe7e94df5dfae: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250903 14:27:28.420928 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.420756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.0.0.1:37902)
0903 14:27:28.420839 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.420875 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 14:27:28.420929 27792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250903 14:27:28.421074 27046 consensus_peers.cc:489] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae -> Peer 55f880c1f5e3421ab6ec375c6d4932d1 (127.19.65.190:37367): Couldn't send request to peer 55f880c1f5e3421ab6ec375c6d4932d1. Status: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:27:28.422082 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a76ae20d7c904e3ea2ebe7e94df5dfae. Latest consensus state: current_term: 3 leader_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:28.422191 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:28.423043 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:28.423130 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:28.423418 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55f880c1f5e3421ab6ec375c6d4932d1" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37367 } } peers { permanent_uuid: "a76ae20d7c904e3ea2ebe7e94df5dfae" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 36775 } attrs { promote: false } } peers { permanent_uuid: "9f0c7257d43944b6b75d66a871ac979e" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46023 } attrs { promote: false } } }
I20250903 14:27:28.423506 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0c7257d43944b6b75d66a871ac979e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:27:28.423671 27800 mvcc.cc:204] Tried to move back new op lower bound from 7196301919926247424 to 7196301919633838080. Current Snapshot: MvccSnapshot[applied={T|T < 7196301919926247424}]
I20250903 14:27:28.430864 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.427903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:43183 (local address 127.0.0.1:33800)
0903 14:27:28.428007 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.428021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:28.428095 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:28.428786 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:28.428788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:28.428801 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:28.429006 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.429012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.429838 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.429841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:28.430527 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.430534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.430625 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.430641 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 14:27:28.430685 (+ 44us) client_negotiation.cc:240] Negotiation successful
0903 14:27:28.430738 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20250903 14:27:28.431409 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.428155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:38077 (local address 127.0.0.1:33928)
0903 14:27:28.428617 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.428629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:28.428701 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:28.429195 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:28.429197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:28.429211 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:28.429351 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.429355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.430245 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.430248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:28.431107 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.431115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.431209 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.431224 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:27:28.431267 (+ 43us) client_negotiation.cc:240] Negotiation successful
0903 14:27:28.431311 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20250903 14:27:28.431742 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.428389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:39959 (local address 127.0.0.1:53532)
0903 14:27:28.429340 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.429351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:28.429424 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:28.429665 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:28.429668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:28.429682 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:28.429883 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.429891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.430688 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.430691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:28.431444 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.431454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.431549 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.431566 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 14:27:28.431614 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:27:28.431663 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":18,"thread_start_us":74,"threads_started":1}
I20250903 14:27:28.432049 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.428366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.19.65.131:38077)
0903 14:27:28.428994 (+ 628us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.428997 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.429014 (+ 17us) server_negotiation.cc:408] Connection header received
0903 14:27:28.429050 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.429052 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.429105 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.429183 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:28.429487 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.430120 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.431233 (+ 1113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.431778 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.431814 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.431865 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.431928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":63,"threads_started":1}
I20250903 14:27:28.432171 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.428080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.19.65.130:43183)
0903 14:27:28.428589 (+ 509us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.428594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.428608 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:28.428651 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.428653 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.428697 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.428782 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:28.429108 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.429735 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.430625 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.431798 (+ 1173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.431936 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.431977 (+ 41us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.432049 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":78,"threads_started":1}
I20250903 14:27:28.432504 27649 tablet_service.cc:1468] Processing CreateTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 (DEFAULT_TABLE table=test-table [id=1bd1afeaddbe4acba21a3995714d3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:28.432591 27512 tablet_service.cc:1468] Processing CreateTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 (DEFAULT_TABLE table=test-table [id=1bd1afeaddbe4acba21a3995714d3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:28.432709 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.428531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.19.65.129:39959)
0903 14:27:28.428720 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.428727 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.429420 (+ 693us) server_negotiation.cc:408] Connection header received
0903 14:27:28.429527 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.429530 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.429578 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.429650 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:28.429977 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.430582 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.431880 (+ 1298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.432478 (+ 598us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.432508 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.432552 (+ 44us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.432601 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
I20250903 14:27:28.432772 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c57641025a4dd5b541c5e5d1b14323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:28.432850 27512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c57641025a4dd5b541c5e5d1b14323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:28.433199 27376 tablet_service.cc:1468] Processing CreateTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 (DEFAULT_TABLE table=test-table [id=1bd1afeaddbe4acba21a3995714d3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:28.433439 27376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c57641025a4dd5b541c5e5d1b14323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:28.434736 27810 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Bootstrap starting.
I20250903 14:27:28.435200 27811 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Bootstrap starting.
I20250903 14:27:28.435644 27810 tablet_bootstrap.cc:654] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:28.435729 27812 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: Bootstrap starting.
I20250903 14:27:28.436219 27811 tablet_bootstrap.cc:654] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:28.436712 27812 tablet_bootstrap.cc:654] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:28.437201 27811 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: No bootstrap required, opened a new log
I20250903 14:27:28.437254 27810 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: No bootstrap required, opened a new log
I20250903 14:27:28.437294 27811 ts_tablet_manager.cc:1397] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:28.437352 27810 ts_tablet_manager.cc:1397] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250903 14:27:28.437793 27812 tablet_bootstrap.cc:492] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: No bootstrap required, opened a new log
I20250903 14:27:28.437757 27811 raft_consensus.cc:357] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.437855 27811 raft_consensus.cc:383] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:28.437812 27810 raft_consensus.cc:357] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.437898 27810 raft_consensus.cc:383] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:28.437944 27810 raft_consensus.cc:738] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99be9c94fbc410ebf5f1c2f503e59b0, State: Initialized, Role: FOLLOWER
I20250903 14:27:28.437947 27812 ts_tablet_manager.cc:1397] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:28.438015 27811 raft_consensus.cc:738] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fea8909934484380f011507ceb0a66, State: Initialized, Role: FOLLOWER
I20250903 14:27:28.438074 27810 consensus_queue.cc:260] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.438151 27811 consensus_queue.cc:260] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.438423 27810 ts_tablet_manager.cc:1428] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:28.438411 27812 raft_consensus.cc:357] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.438458 27811 ts_tablet_manager.cc:1428] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:28.438488 27812 raft_consensus.cc:383] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:28.438513 27812 raft_consensus.cc:738] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b407477dde04cc58b0049f4251e8073, State: Initialized, Role: FOLLOWER
I20250903 14:27:28.438627 27812 consensus_queue.cc:260] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.438915 27812 ts_tablet_manager.cc:1428] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:28.439042 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.438794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:34787)
0903 14:27:28.438927 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.438972 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20250903 14:27:28.439747 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.439244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:44799)
0903 14:27:28.439624 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.439667 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20250903 14:27:28.440461 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.440128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:47189)
0903 14:27:28.440358 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.440396 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20250903 14:27:28.440882 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.439608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:37341)
0903 14:27:28.440777 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.440822 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1100}
I20250903 14:27:28.442039 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.441664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:46737)
0903 14:27:28.441934 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.441975 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20250903 14:27:28.442397 27817 raft_consensus.cc:491] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:28.442456 27717 heartbeater.cc:499] Master 127.19.65.189:36775 was elected leader, sending a full tablet report...
I20250903 14:27:28.442524 27817 raft_consensus.cc:513] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.442728 27580 heartbeater.cc:499] Master 127.19.65.189:36775 was elected leader, sending a full tablet report...
I20250903 14:27:28.443961 27817 leader_election.cc:290] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183), 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:28.444209 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.443922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:60763)
0903 14:27:28.444101 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.444142 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20250903 14:27:28.444693 27444 heartbeater.cc:499] Master 127.19.65.189:36775 was elected leader, sending a full tablet report...
I20250903 14:27:28.446748 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.443606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:43183 (local address 127.19.65.131:47543)
0903 14:27:28.443876 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.443888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:28.443975 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:28.444297 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:28.444300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:28.444322 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:28.444535 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.444545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.445791 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.445797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:28.446415 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.446422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.446541 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.446557 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 14:27:28.446601 (+ 44us) client_negotiation.cc:240] Negotiation successful
0903 14:27:28.446651 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20250903 14:27:28.446909 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.443631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:39959 (local address 127.19.65.131:42407)
0903 14:27:28.444318 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.444329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:28.444442 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:28.444710 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:28.444712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:28.444728 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:28.444949 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.444955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.445897 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.445899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:28.446474 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:28.446481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.446626 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.446643 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 14:27:28.446692 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 14:27:28.446744 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":77,"threads_started":1}
I20250903 14:27:28.447562 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.443775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:47543 (local address 127.19.65.130:43183)
0903 14:27:28.443896 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.443899 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.443937 (+ 38us) server_negotiation.cc:408] Connection header received
0903 14:27:28.444088 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.444091 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.444132 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.444192 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:28.444986 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.445656 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.446967 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.447353 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.447386 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.447433 (+ 47us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.447476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:27:28.447606 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.444010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:42407 (local address 127.19.65.129:39959)
0903 14:27:28.444133 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.444137 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.444385 (+ 248us) server_negotiation.cc:408] Connection header received
0903 14:27:28.444559 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.444561 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.444631 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.444704 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:28.445057 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.445744 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.446638 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.447390 (+ 752us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.447420 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.447462 (+ 42us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.447506 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:28.448158 27532 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16c57641025a4dd5b541c5e5d1b14323" candidate_uuid: "f8fea8909934484380f011507ceb0a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" is_pre_election: true
I20250903 14:27:28.448331 27532 raft_consensus.cc:2466] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8fea8909934484380f011507ceb0a66 in term 0.
I20250903 14:27:28.448621 27396 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16c57641025a4dd5b541c5e5d1b14323" candidate_uuid: "f8fea8909934484380f011507ceb0a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b407477dde04cc58b0049f4251e8073" is_pre_election: true
I20250903 14:27:28.448733 27396 raft_consensus.cc:2466] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8fea8909934484380f011507ceb0a66 in term 0.
I20250903 14:27:28.448837 27604 leader_election.cc:304] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d99be9c94fbc410ebf5f1c2f503e59b0, f8fea8909934484380f011507ceb0a66; no voters:
I20250903 14:27:28.449165 27817 raft_consensus.cc:2802] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:28.449226 27817 raft_consensus.cc:491] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:28.449285 27817 raft_consensus.cc:3058] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:28.450071 27817 raft_consensus.cc:513] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.450330 27817 leader_election.cc:290] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [CANDIDATE]: Term 1 election: Requested vote from peers d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183), 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959)
I20250903 14:27:28.450603 27532 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16c57641025a4dd5b541c5e5d1b14323" candidate_uuid: "f8fea8909934484380f011507ceb0a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0"
I20250903 14:27:28.450603 27396 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16c57641025a4dd5b541c5e5d1b14323" candidate_uuid: "f8fea8909934484380f011507ceb0a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b407477dde04cc58b0049f4251e8073"
I20250903 14:27:28.450704 27396 raft_consensus.cc:3058] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:28.450704 27532 raft_consensus.cc:3058] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:28.451591 27532 raft_consensus.cc:2466] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fea8909934484380f011507ceb0a66 in term 1.
I20250903 14:27:28.451591 27396 raft_consensus.cc:2466] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fea8909934484380f011507ceb0a66 in term 1.
I20250903 14:27:28.451841 27603 leader_election.cc:304] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b407477dde04cc58b0049f4251e8073, f8fea8909934484380f011507ceb0a66; no voters:
I20250903 14:27:28.452005 27817 raft_consensus.cc:2802] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:28.452184 27817 raft_consensus.cc:695] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 1 LEADER]: Becoming Leader. State: Replica: f8fea8909934484380f011507ceb0a66, State: Running, Role: LEADER
I20250903 14:27:28.452364 27817 consensus_queue.cc:237] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } }
I20250903 14:27:28.452666 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.452474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:50567)
0903 14:27:28.452569 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.452613 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20250903 14:27:28.454145 27060 catalog_manager.cc:5582] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8fea8909934484380f011507ceb0a66 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "f8fea8909934484380f011507ceb0a66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8fea8909934484380f011507ceb0a66" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 } health_report { overall_health: UNKNOWN } } }
I20250903 14:27:28.463595 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.461320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.19.65.131:38077)
0903 14:27:28.461402 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:28.461405 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:28.461417 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:28.461581 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:28.461583 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:28.461624 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:28.461693 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:28.461939 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.462344 (+ 405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:28.462842 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:28.462955 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:28.462994 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:28.463381 (+ 387us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:28.463461 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:28.463498 (+ 37us) server_negotiation.cc:300] Negotiation successful
0903 14:27:28.463530 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20250903 14:27:28.465791 27396 raft_consensus.cc:1273] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Refusing update from remote peer f8fea8909934484380f011507ceb0a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:28.465950 27532 raft_consensus.cc:1273] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Refusing update from remote peer f8fea8909934484380f011507ceb0a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:28.466215 27820 consensus_queue.cc:1035] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b407477dde04cc58b0049f4251e8073" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:28.466403 27817 consensus_queue.cc:1035] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99be9c94fbc410ebf5f1c2f503e59b0" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:28.467440 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.467217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:42579)
0903 14:27:28.467330 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.467380 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20250903 14:27:28.468900 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.468716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:47961)
0903 14:27:28.468797 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.468841 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20250903 14:27:28.470058 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.469734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:37067)
0903 14:27:28.469955 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.470001 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20250903 14:27:28.470546 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.470268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.129:60847)
0903 14:27:28.470431 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.470479 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20250903 14:27:28.470942 27823 mvcc.cc:204] Tried to move back new op lower bound from 7196301920111685632 to 7196301920062881792. Current Snapshot: MvccSnapshot[applied={T|T < 7196301920111685632}]
I20250903 14:27:28.472965 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.472780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.130:50231)
0903 14:27:28.472862 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.472905 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20250903 14:27:28.473158 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:28.472788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37367 (local address 127.19.65.131:43071)
0903 14:27:28.473035 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.473102 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20250903 14:27:28.478339 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34670:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:28.478439 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34670:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:28.481758 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a76ae20d7c904e3ea2ebe7e94df5dfae: Sending DeleteTablet for 3 replicas of tablet 16c57641025a4dd5b541c5e5d1b14323
I20250903 14:27:28.482280 27649 tablet_service.cc:1515] Processing DeleteTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:28 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:33928
I20250903 14:27:28.482367 27512 tablet_service.cc:1515] Processing DeleteTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:28 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:33800
I20250903 14:27:28.482522 27376 tablet_service.cc:1515] Processing DeleteTablet for tablet 16c57641025a4dd5b541c5e5d1b14323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:28 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:53532
I20250903 14:27:28.482637 27833 tablet_replica.cc:331] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: stopping tablet replica
I20250903 14:27:28.482743 27833 raft_consensus.cc:2241] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:28.482776 27834 tablet_replica.cc:331] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073: stopping tablet replica
I20250903 14:27:28.482843 27833 raft_consensus.cc:2270] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:28.482853 27834 raft_consensus.cc:2241] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:28.482944 27834 raft_consensus.cc:2270] T 16c57641025a4dd5b541c5e5d1b14323 P 5b407477dde04cc58b0049f4251e8073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:28.483307 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27313
I20250903 14:27:28.483953 27833 ts_tablet_manager.cc:1905] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:28.485590 27833 ts_tablet_manager.cc:1918] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:28.485639 27833 log.cc:1199] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/16c57641025a4dd5b541c5e5d1b14323
I20250903 14:27:28.485850 27833 ts_tablet_manager.cc:1939] T 16c57641025a4dd5b541c5e5d1b14323 P d99be9c94fbc410ebf5f1c2f503e59b0: Deleting consensus metadata
I20250903 14:27:28.486318 27047 catalog_manager.cc:4928] TS d99be9c94fbc410ebf5f1c2f503e59b0 (127.19.65.130:43183): tablet 16c57641025a4dd5b541c5e5d1b14323 (table test-table [id=1bd1afeaddbe4acba21a3995714d3049]) successfully deleted
I20250903 14:27:28.489626 27832 tablet_replica.cc:331] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: stopping tablet replica
I20250903 14:27:28.489734 27832 raft_consensus.cc:2241] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:28.489962 27832 raft_consensus.cc:2270] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:28.490446 27832 ts_tablet_manager.cc:1905] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:28.492095 27832 ts_tablet_manager.cc:1918] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:28.492159 27832 log.cc:1199] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/16c57641025a4dd5b541c5e5d1b14323
I20250903 14:27:28.492445 27832 ts_tablet_manager.cc:1939] T 16c57641025a4dd5b541c5e5d1b14323 P f8fea8909934484380f011507ceb0a66: Deleting consensus metadata
W20250903 14:27:28.492956 27808 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:28.492690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:39959 (local address 127.0.0.1:53542)
0903 14:27:28.492832 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:28.492877 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:39959: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":79}
I20250903 14:27:28.493046 27045 catalog_manager.cc:4928] TS f8fea8909934484380f011507ceb0a66 (127.19.65.131:38077): tablet 16c57641025a4dd5b541c5e5d1b14323 (table test-table [id=1bd1afeaddbe4acba21a3995714d3049]) successfully deleted
W20250903 14:27:28.493090 27046 catalog_manager.cc:4671] TS 5b407477dde04cc58b0049f4251e8073 (127.19.65.129:39959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16c57641025a4dd5b541c5e5d1b14323: Network error: Client connection negotiation failed: client connection to 127.19.65.129:39959: connect: Connection reset by peer (error 104)
I20250903 14:27:28.493402 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27449
I20250903 14:27:28.500402 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27585
I20250903 14:27:28.505563 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 26927
I20250903 14:27:28.510604 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27117
2025-09-03T14:27:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8310 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:28 dist-test-slave-4wpb krb5kdc[27837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:28 dist-test-slave-4wpb krb5kdc[27837](info): set up 2 sockets
Sep 03 14:27:28 dist-test-slave-4wpb krb5kdc[27837](info): commencing operation
krb5kdc: starting...
W20250903 14:27:30.277957 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.745s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:30.385458 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34745
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:30.475474 27853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:30.475653 27853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:30.475731 27853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:30.478368 27853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:30.478415 27853 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 14:27:30.478441 27853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:30.478475 27853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:30.478508 27853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:30.478538 27853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:30.481928 27853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:34745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:30.482692 27853 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:30.483342 27853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:30.488133 27861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:30.488118 27859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:30.488121 27858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:30.488276 27853 server_base.cc:1047] running on GCE node
I20250903 14:27:30.488791 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:30.489563 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:30.490693 27853 hybrid_clock.cc:648] HybridClock initialized: now 1756909650490684 us; error 27 us; skew 500 ppm
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:30.492901 27853 init.cc:377] Logged in from keytab as oryx/127.19.65.190@KRBTEST.COM (short username oryx)
I20250903 14:27:30.493914 27853 webserver.cc:480] Webserver started at http://127.19.65.190:37339/ using document root <none> and password file <none>
I20250903 14:27:30.494390 27853 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:30.494448 27853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:30.494628 27853 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:30.496688 27853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "3f03fdc1b79e49ee86ac114b551af106"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "b8925ea0b75a78a0409218f914a426bf"
server_key_iv: "4f2c2e94a67b9da4a01f788340bb50b3"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.497114 27853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "3f03fdc1b79e49ee86ac114b551af106"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "b8925ea0b75a78a0409218f914a426bf"
server_key_iv: "4f2c2e94a67b9da4a01f788340bb50b3"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.501327 27853 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 14:27:30.505760 27868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:30.508316 27853 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250903 14:27:30.508435 27853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "3f03fdc1b79e49ee86ac114b551af106"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "b8925ea0b75a78a0409218f914a426bf"
server_key_iv: "4f2c2e94a67b9da4a01f788340bb50b3"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.508531 27853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:30.529310 27853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:30.531924 27853 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:30.532071 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:30.538820 27930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:34745 every 8 connection(s)
I20250903 14:27:30.538826 27853 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:34745
I20250903 14:27:30.539656 27853 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:30.541689 27931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:30.546360 27931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Bootstrap starting.
I20250903 14:27:30.548128 27931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:30.548660 27931 log.cc:826] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:30.549372 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27853
I20250903 14:27:30.549464 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:30.549682 19718 external_mini_cluster.cc:1468] Setting key 92b8748a9d70528a6ab832d33e8e0c95
I20250903 14:27:30.550132 27931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: No bootstrap required, opened a new log
I20250903 14:27:30.552320 27931 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } }
I20250903 14:27:30.552471 27931 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:30.552523 27931 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f03fdc1b79e49ee86ac114b551af106, State: Initialized, Role: FOLLOWER
I20250903 14:27:30.552861 27931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } }
I20250903 14:27:30.552976 27931 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:30.553035 27931 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:30.553123 27931 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:30.553815 27931 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } }
I20250903 14:27:30.554083 27931 leader_election.cc:304] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f03fdc1b79e49ee86ac114b551af106; no voters:
I20250903 14:27:30.554311 27931 leader_election.cc:290] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:30.554394 27936 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:30.554550 27936 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [term 1 LEADER]: Becoming Leader. State: Replica: 3f03fdc1b79e49ee86ac114b551af106, State: Running, Role: LEADER
I20250903 14:27:30.554823 27936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } }
I20250903 14:27:30.555259 27931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:30.555994 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f03fdc1b79e49ee86ac114b551af106. Latest consensus state: current_term: 1 leader_uuid: "3f03fdc1b79e49ee86ac114b551af106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } } }
I20250903 14:27:30.556107 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:30.556315 27937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f03fdc1b79e49ee86ac114b551af106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f03fdc1b79e49ee86ac114b551af106" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34745 } } }
I20250903 14:27:30.556389 27937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:30.556670 27944 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:30.559358 27944 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:30.562781 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.550464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.19.65.190:34745)
0903 14:27:30.550773 (+ 309us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:30.550781 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:30.550806 (+ 25us) server_negotiation.cc:408] Connection header received
0903 14:27:30.552009 (+ 1203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:30.552021 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:30.552251 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:30.552495 (+ 244us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:30.553356 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.554111 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.554899 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.555164 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.558483 (+ 3319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:30.558502 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:30.558514 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:30.558547 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:30.560467 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.560878 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.560883 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.560887 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.560960 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.561189 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.561192 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.561194 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.561560 (+ 366us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:30.561738 (+ 178us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:30.561965 (+ 227us) server_negotiation.cc:300] Negotiation successful
0903 14:27:30.562175 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20250903 14:27:30.563910 27944 catalog_manager.cc:1349] Generated new cluster ID: ccc85c329a55445389ea95e7bc7148e2
I20250903 14:27:30.563966 27944 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:30.583873 27944 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:30.584784 27944 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:30.594955 27944 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Generated new TSK 0
I20250903 14:27:30.595535 27944 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:30.637660 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34745
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:30.727475 27959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:30.727624 27959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:30.727691 27959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:30.730472 27959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:30.730520 27959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:30.730566 27959 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:27:30.730621 27959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:30.734004 27959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:30.734763 27959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:30.735473 27959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:30.740687 27967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:30.740813 27964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:30.740854 27959 server_base.cc:1047] running on GCE node
W20250903 14:27:30.740717 27965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:30.741237 27959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:30.741698 27959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:30.742830 27959 hybrid_clock.cc:648] HybridClock initialized: now 1756909650742820 us; error 22 us; skew 500 ppm
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:30.745013 27959 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20250903 14:27:30.745921 27959 webserver.cc:480] Webserver started at http://127.19.65.129:43491/ using document root <none> and password file <none>
I20250903 14:27:30.746402 27959 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:30.746448 27959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:30.746600 27959 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:30.748008 27959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "f356a5927e614fbab65e145d538b27e8"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "56c489502c3efcfba29a30b62fe3ae4a"
server_key_iv: "28e0880aee187751782c2e4d2e5b1b0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.748390 27959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "f356a5927e614fbab65e145d538b27e8"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "56c489502c3efcfba29a30b62fe3ae4a"
server_key_iv: "28e0880aee187751782c2e4d2e5b1b0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.751030 27959 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:27:30.752851 27974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:30.753710 27959 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:30.753798 27959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "f356a5927e614fbab65e145d538b27e8"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "56c489502c3efcfba29a30b62fe3ae4a"
server_key_iv: "28e0880aee187751782c2e4d2e5b1b0b"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.753870 27959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:30.767954 27959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:30.769977 27959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:30.770123 27959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:30.771203 27959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:30.771245 27959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:30.771291 27959 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:30.771320 27959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:30.780882 27959 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42367
I20250903 14:27:30.780961 28089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:42367 every 8 connection(s)
I20250903 14:27:30.781888 27959 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:30.782446 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.771751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:56937 (local address 127.19.65.190:34745)
0903 14:27:30.771851 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:30.771855 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:30.772535 (+ 680us) server_negotiation.cc:408] Connection header received
0903 14:27:30.773273 (+ 738us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:30.773277 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:30.773343 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:30.773424 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:30.775040 (+ 1616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.775825 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.776723 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.776948 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.778901 (+ 1953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:30.778924 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:30.778928 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:30.778958 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:30.780650 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.781161 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.781165 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.781166 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.781233 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.781508 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.781511 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.781513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.781691 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:30.781802 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:30.782208 (+ 406us) server_negotiation.cc:300] Negotiation successful
0903 14:27:30.782330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250903 14:27:30.782977 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.771991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.129:56937)
0903 14:27:30.772416 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:30.772449 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:30.773099 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:30.773622 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:30.773634 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:30.774133 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:30.774854 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.774869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.776000 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.776003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:30.776562 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.776574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.776741 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.777235 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:30.777276 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:30.778754 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:30.780753 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.780758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.780766 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.781074 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.781326 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.781328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.781330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.781395 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.781775 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:30.781778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:30.781981 (+ 203us) client_negotiation.cc:769] Sending connection context
0903 14:27:30.782137 (+ 156us) client_negotiation.cc:240] Negotiation successful
0903 14:27:30.782352 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20250903 14:27:30.784013 28090 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34745
I20250903 14:27:30.784137 28090 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:30.784544 28090 heartbeater.cc:507] Master 127.19.65.190:34745 requested a full tablet report, sending...
I20250903 14:27:30.785871 27884 ts_manager.cc:194] Registered new tserver with Master: f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367)
I20250903 14:27:30.787022 27884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:56937
I20250903 14:27:30.790536 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.785633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.129:48397)
0903 14:27:30.785864 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:30.785878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:30.785985 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:30.786200 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:30.786203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:30.786360 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:30.786568 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.786575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.787208 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.787210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:30.787629 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.787638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.787737 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.788173 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:30.788185 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:30.788416 (+ 231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:30.789712 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.789714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.789716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.789849 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.790002 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.790004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.790005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.790036 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.790236 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:30.790238 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:30.790294 (+ 56us) client_negotiation.cc:769] Sending connection context
0903 14:27:30.790359 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 14:27:30.790434 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20250903 14:27:30.790546 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.785691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:48397 (local address 127.19.65.190:34745)
0903 14:27:30.785813 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:30.785817 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:30.785954 (+ 137us) server_negotiation.cc:408] Connection header received
0903 14:27:30.786083 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:30.786085 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:30.786116 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:30.786172 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:30.786694 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.787085 (+ 391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.787732 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.787895 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.788524 (+ 629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:30.788538 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:30.788541 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:30.788562 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:30.789632 (+ 1070us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.789918 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.789921 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.789922 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.789951 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.790083 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.790084 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.790085 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.790175 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:30.790231 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:30.790393 (+ 162us) server_negotiation.cc:300] Negotiation successful
0903 14:27:30.790461 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 14:27:30.791471 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 27959
I20250903 14:27:30.791550 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:30.791749 19718 external_mini_cluster.cc:1468] Setting key 7ceea37a0614d6d188b01a9c05c98460
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:30.828040 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34745
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:30.919188 28101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:30.919327 28101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:30.919374 28101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:30.922290 28101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:30.922335 28101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:30.922377 28101 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:27:30.922415 28101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:30.925828 28101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:30.926568 28101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:30.927184 28101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:30.932125 28106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:30.932134 28107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:30.932192 28109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:30.932261 28101 server_base.cc:1047] running on GCE node
I20250903 14:27:30.932631 28101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:30.933070 28101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:30.934203 28101 hybrid_clock.cc:648] HybridClock initialized: now 1756909650934194 us; error 28 us; skew 500 ppm
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:30.936509 28101 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20250903 14:27:30.937276 28101 webserver.cc:480] Webserver started at http://127.19.65.130:46199/ using document root <none> and password file <none>
I20250903 14:27:30.937729 28101 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:30.937775 28101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:30.937916 28101 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:30.939227 28101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "3aa25bdfbcba426a8348dcfce02037d9"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "32a4e975163fef6df408bd1c9243b763"
server_key_iv: "b0c561bd061a78c2a05034def5d74068"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.939620 28101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "3aa25bdfbcba426a8348dcfce02037d9"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "32a4e975163fef6df408bd1c9243b763"
server_key_iv: "b0c561bd061a78c2a05034def5d74068"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.942317 28101 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250903 14:27:30.943991 28116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:30.944774 28101 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:30.944873 28101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "3aa25bdfbcba426a8348dcfce02037d9"
format_stamp: "Formatted at 2025-09-03 14:27:30 on dist-test-slave-4wpb"
server_key: "32a4e975163fef6df408bd1c9243b763"
server_key_iv: "b0c561bd061a78c2a05034def5d74068"
server_key_version: "encryptionkey@0"
I20250903 14:27:30.944937 28101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:30.962302 28101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:30.964169 28101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:30.964288 28101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:30.965346 28101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:30.965391 28101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:30.965425 28101 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:30.965457 28101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:27:30 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:30.974987 28101 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:33293
I20250903 14:27:30.975678 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.966036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:39067 (local address 127.19.65.190:34745)
0903 14:27:30.966149 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:30.966152 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:30.966735 (+ 583us) server_negotiation.cc:408] Connection header received
0903 14:27:30.967430 (+ 695us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:30.967435 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:30.967498 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:30.967584 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:30.968727 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.969311 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.970176 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.970378 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.972365 (+ 1987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:30.972381 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:30.972383 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:30.972406 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:30.973935 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.974384 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.974387 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.974389 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.974426 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.974653 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.974656 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.974657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.974847 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:30.974952 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:30.975472 (+ 520us) server_negotiation.cc:300] Negotiation successful
0903 14:27:30.975567 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 14:27:30.975946 28101 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:30.976341 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.966272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.130:39067)
0903 14:27:30.966624 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:30.966652 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:30.967229 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:30.967718 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:30.967726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:30.968105 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:30.968594 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.968605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.969453 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.969459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:30.970057 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.970068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.970245 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.970759 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:30.970790 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:30.972237 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:30.974041 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.974046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.974055 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.974299 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.974499 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.974502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.974504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.974580 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.974983 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:30.974988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:30.975223 (+ 235us) client_negotiation.cc:769] Sending connection context
0903 14:27:30.975419 (+ 196us) client_negotiation.cc:240] Negotiation successful
0903 14:27:30.975642 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
I20250903 14:27:30.976382 28231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:33293 every 8 connection(s)
I20250903 14:27:30.977505 28232 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34745
I20250903 14:27:30.977623 28232 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:30.977988 28232 heartbeater.cc:507] Master 127.19.65.190:34745 requested a full tablet report, sending...
I20250903 14:27:30.978768 27885 ts_manager.cc:194] Registered new tserver with Master: 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293)
I20250903 14:27:30.979354 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:39067
I20250903 14:27:30.981884 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28101
I20250903 14:27:30.982018 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:30.982290 19718 external_mini_cluster.cc:1468] Setting key 188ec35f3c15c547de229736b8699d49
I20250903 14:27:30.985174 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.979076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.130:33193)
0903 14:27:30.979321 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:30.979331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:30.979409 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:30.979651 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:30.979654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:30.979805 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:30.979959 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.979964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.980788 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.980790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:30.981104 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:30.981109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.981184 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.981622 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:30.981633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:30.981845 (+ 212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:30.983908 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.983914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.983918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.984114 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.984296 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:30.984301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:30.984305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:30.984356 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:30.984601 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:30.984606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:30.984683 (+ 77us) client_negotiation.cc:769] Sending connection context
0903 14:27:30.984898 (+ 215us) client_negotiation.cc:240] Negotiation successful
0903 14:27:30.985044 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20250903 14:27:30.985285 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:30.979122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33193 (local address 127.19.65.190:34745)
0903 14:27:30.979210 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:30.979213 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:30.979389 (+ 176us) server_negotiation.cc:408] Connection header received
0903 14:27:30.979518 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:30.979522 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:30.979565 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:30.979628 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:30.980059 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.980702 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:30.981185 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:30.981299 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:30.981948 (+ 649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:30.981965 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:30.981969 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:30.982014 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:30.983808 (+ 1794us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.984196 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.984198 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.984199 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.984228 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:30.984420 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:30.984422 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:30.984423 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:30.984535 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:30.984594 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:30.984973 (+ 379us) server_negotiation.cc:300] Negotiation successful
0903 14:27:30.985097 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab.
I20250903 14:27:31.018033 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:34745
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:31.105330 28243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:31.105540 28243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:31.105624 28243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:31.108258 28243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:31.108321 28243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:31.108371 28243 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 14:27:31.108445 28243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:31.111995 28243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:31.112720 28243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:31.113397 28243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:31.118902 28248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:31.119032 28243 server_base.cc:1047] running on GCE node
W20250903 14:27:31.118899 28251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:31.118940 28249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:31.119453 28243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:31.119917 28243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:31.121049 28243 hybrid_clock.cc:648] HybridClock initialized: now 1756909651121030 us; error 32 us; skew 500 ppm
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:31.123391 28243 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20250903 14:27:31.124164 28243 webserver.cc:480] Webserver started at http://127.19.65.131:39295/ using document root <none> and password file <none>
I20250903 14:27:31.124574 28243 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:31.124619 28243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:31.124783 28243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:31.126277 28243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "24c2eb0cfcbe45d29515257222ee617f"
format_stamp: "Formatted at 2025-09-03 14:27:31 on dist-test-slave-4wpb"
server_key: "b7b4d25f324aef9d1df0c7604bfc960a"
server_key_iv: "325d01efb4bead079ea029b24b7fe5fa"
server_key_version: "encryptionkey@0"
I20250903 14:27:31.126648 28243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "24c2eb0cfcbe45d29515257222ee617f"
format_stamp: "Formatted at 2025-09-03 14:27:31 on dist-test-slave-4wpb"
server_key: "b7b4d25f324aef9d1df0c7604bfc960a"
server_key_iv: "325d01efb4bead079ea029b24b7fe5fa"
server_key_version: "encryptionkey@0"
I20250903 14:27:31.129201 28243 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:27:31.130868 28258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:31.131718 28243 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:31.131798 28243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "24c2eb0cfcbe45d29515257222ee617f"
format_stamp: "Formatted at 2025-09-03 14:27:31 on dist-test-slave-4wpb"
server_key: "b7b4d25f324aef9d1df0c7604bfc960a"
server_key_iv: "325d01efb4bead079ea029b24b7fe5fa"
server_key_version: "encryptionkey@0"
I20250903 14:27:31.131857 28243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:31.152917 28243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:31.155377 28243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:31.155505 28243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:31.156800 28243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:31.156844 28243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:31.156877 28243 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:31.156909 28243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:31.166292 28243 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:42063
I20250903 14:27:31.166327 28373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:42063 every 8 connection(s)
I20250903 14:27:31.167132 28243 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:31.167773 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.157320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:50255 (local address 127.19.65.190:34745)
0903 14:27:31.157429 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.157432 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.158062 (+ 630us) server_negotiation.cc:408] Connection header received
0903 14:27:31.158702 (+ 640us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.158706 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.158761 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.158838 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.160347 (+ 1509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.161005 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.161819 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.162012 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.164337 (+ 2325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:31.164360 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:31.164366 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:31.164395 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:31.166215 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.166637 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.166641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.166642 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.166689 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.166897 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.166900 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.166902 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.167065 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:31.167164 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.167535 (+ 371us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.167633 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:27:31.168267 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.157597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.131:50255)
0903 14:27:31.157946 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.157974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.158541 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.159016 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.159027 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.159499 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:31.160215 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.160229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.161130 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.161135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.161713 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.161723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.161909 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.162467 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:31.162498 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:31.164175 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:31.166323 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.166328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.166338 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.166549 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.166763 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.166765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.166767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.166824 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.167150 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:31.167153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:31.167340 (+ 187us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.167486 (+ 146us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.167682 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20250903 14:27:31.169301 28374 heartbeater.cc:344] Connected to a master server at 127.19.65.190:34745
I20250903 14:27:31.169426 28374 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:31.169828 28374 heartbeater.cc:507] Master 127.19.65.190:34745 requested a full tablet report, sending...
I20250903 14:27:31.170684 27884 ts_manager.cc:194] Registered new tserver with Master: 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.171273 27884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:50255
I20250903 14:27:31.171912 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28243
I20250903 14:27:31.172012 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:31.172227 19718 external_mini_cluster.cc:1468] Setting key 9d9ef8751860c5b737daed4a61d6bc20
I20250903 14:27:31.174100 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:27:31.176637 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.170974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59399 (local address 127.19.65.190:34745)
0903 14:27:31.171075 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.171079 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.171241 (+ 162us) server_negotiation.cc:408] Connection header received
0903 14:27:31.171349 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.171351 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.171392 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.171456 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.171949 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.172406 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.173099 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.173320 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.173881 (+ 561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:31.173895 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:31.173898 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:31.173920 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:31.175337 (+ 1417us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.175694 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.175699 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.175704 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.175752 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.175940 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.175946 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.175950 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.176133 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:31.176218 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.176466 (+ 248us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.176544 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:27:31.176806 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.170917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.19.65.131:59399)
0903 14:27:31.171169 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.171181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.171277 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.171487 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.171493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.171668 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:31.171851 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.171857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.172493 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.172496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.172941 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.172952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.173113 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.173541 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:31.173553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:31.173777 (+ 224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:31.175447 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.175450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.175452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.175609 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.175829 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.175831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.175832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.175869 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.176223 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:31.176228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:31.176316 (+ 88us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.176432 (+ 116us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.176568 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20250903 14:27:31.191987 27935 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:31.187941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.19.65.190:34745)
0903 14:27:31.188030 (+ 89us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.188033 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.188042 (+ 9us) server_negotiation.cc:408] Connection header received
0903 14:27:31.188065 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.188067 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.188093 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.188148 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.188733 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.189130 (+ 397us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.189678 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.189775 (+ 97us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.191897 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58596: BlockingRecv error: recv got EOF from 127.0.0.1:58596 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:31.198932 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.193102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.19.65.190:34745)
0903 14:27:31.193212 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.193216 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.193284 (+ 68us) server_negotiation.cc:408] Connection header received
0903 14:27:31.193382 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.193385 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.193417 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.193504 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.194078 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.194471 (+ 393us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.194991 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.195181 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.196797 (+ 1616us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:31.196809 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:31.196811 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:31.196828 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:31.197811 (+ 983us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.198121 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.198126 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.198128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.198161 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.198348 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.198351 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.198352 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.198454 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:31.198553 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.198724 (+ 171us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.198812 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:27:31.200837 27884 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:31.202950 27884 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:31.212872 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.209787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42063 (local address 127.0.0.1:59014)
0903 14:27:31.210025 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.210036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.210107 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.210716 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.210722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.210743 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.210963 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.210968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.211849 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.211852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.212490 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.212496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.212595 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.212637 (+ 42us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.212707 (+ 70us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.212774 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20250903 14:27:31.213364 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.209740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42367 (local address 127.0.0.1:46758)
0903 14:27:31.209958 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.209970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.210044 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.210824 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.210828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.210865 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.211030 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.211039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.212221 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.212224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.213044 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.213053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.213143 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.213161 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.213209 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.213260 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"spinlock_wait_cycles":6912,"thread_start_us":98,"threads_started":1}
I20250903 14:27:31.213795 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.210114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.19.65.131:42063)
0903 14:27:31.210354 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.210359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.210374 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:31.210418 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.210422 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.210559 (+ 137us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.211093 (+ 534us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.211201 (+ 108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.211757 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.212609 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.213243 (+ 634us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.213337 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.213389 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.213698 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20250903 14:27:31.214056 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.209273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33293 (local address 127.0.0.1:34468)
0903 14:27:31.209553 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.209586 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.209698 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.210823 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.210825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.210852 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.211033 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.211038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.212329 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.212332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.213723 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.213733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.213836 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.213851 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.213896 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.213944 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"spinlock_wait_cycles":2176,"thread_start_us":55,"threads_started":1}
I20250903 14:27:31.214076 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.209880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.19.65.129:42367)
0903 14:27:31.210308 (+ 428us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.210313 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.210333 (+ 20us) server_negotiation.cc:408] Connection header received
0903 14:27:31.210412 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.210416 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.210560 (+ 144us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.210681 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.211141 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.212112 (+ 971us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.213205 (+ 1093us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.213735 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.213830 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.213886 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.213959 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":83,"threads_started":1}
I20250903 14:27:31.214677 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.209375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.19.65.130:33293)
0903 14:27:31.210430 (+ 1055us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.210438 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.210460 (+ 22us) server_negotiation.cc:408] Connection header received
0903 14:27:31.210537 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.210541 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.210690 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.211154 (+ 464us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.211297 (+ 143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.212239 (+ 942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.213866 (+ 1627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.214337 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.214416 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.214469 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.214548 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":115,"threads_started":1}
I20250903 14:27:31.216172 28024 tablet_service.cc:1468] Processing CreateTablet for tablet 74cfb5f34323411a82426ee83f90a13e (DEFAULT_TABLE table=test-table [id=3a31e345a3c0416c83153457c6622b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.216171 28308 tablet_service.cc:1468] Processing CreateTablet for tablet 74cfb5f34323411a82426ee83f90a13e (DEFAULT_TABLE table=test-table [id=3a31e345a3c0416c83153457c6622b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.216465 28166 tablet_service.cc:1468] Processing CreateTablet for tablet 74cfb5f34323411a82426ee83f90a13e (DEFAULT_TABLE table=test-table [id=3a31e345a3c0416c83153457c6622b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.216961 28024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74cfb5f34323411a82426ee83f90a13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.216961 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74cfb5f34323411a82426ee83f90a13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.217177 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74cfb5f34323411a82426ee83f90a13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.221362 28402 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Bootstrap starting.
I20250903 14:27:31.222364 28404 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Bootstrap starting.
I20250903 14:27:31.223358 28402 tablet_bootstrap.cc:654] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.223724 28404 tablet_bootstrap.cc:654] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.224216 28402 log.cc:826] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:31.224586 28404 log.cc:826] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:31.224885 28403 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Bootstrap starting.
I20250903 14:27:31.226173 28402 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: No bootstrap required, opened a new log
I20250903 14:27:31.226173 28404 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: No bootstrap required, opened a new log
I20250903 14:27:31.226329 28402 ts_tablet_manager.cc:1397] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:27:31.226329 28404 ts_tablet_manager.cc:1397] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:31.226572 28403 tablet_bootstrap.cc:654] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.227458 28403 log.cc:826] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:31.228663 28403 tablet_bootstrap.cc:492] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: No bootstrap required, opened a new log
I20250903 14:27:31.228808 28403 ts_tablet_manager.cc:1397] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:31.228892 28404 raft_consensus.cc:357] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.228893 28402 raft_consensus.cc:357] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.229091 28402 raft_consensus.cc:383] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.229091 28404 raft_consensus.cc:383] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.229151 28404 raft_consensus.cc:738] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.229151 28402 raft_consensus.cc:738] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.229579 28402 consensus_queue.cc:260] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.229579 28404 consensus_queue.cc:260] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.229799 28408 raft_consensus.cc:491] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:31.229969 28408 raft_consensus.cc:513] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.230482 28404 ts_tablet_manager.cc:1428] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250903 14:27:31.230783 28090 heartbeater.cc:499] Master 127.19.65.190:34745 was elected leader, sending a full tablet report...
I20250903 14:27:31.230868 28402 ts_tablet_manager.cc:1428] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250903 14:27:31.231142 28232 heartbeater.cc:499] Master 127.19.65.190:34745 was elected leader, sending a full tablet report...
I20250903 14:27:31.232173 28403 raft_consensus.cc:357] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.232347 28403 raft_consensus.cc:383] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.232393 28403 raft_consensus.cc:738] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.232779 28403 consensus_queue.cc:260] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.233668 28403 ts_tablet_manager.cc:1428] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 14:27:31.233800 28374 heartbeater.cc:499] Master 127.19.65.190:34745 was elected leader, sending a full tablet report...
I20250903 14:27:31.234323 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.231280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33293 (local address 127.19.65.129:44633)
0903 14:27:31.231514 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.231527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.231591 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.231968 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.231970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.231985 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.232192 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.232199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.233105 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.233109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.233955 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.233964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.234066 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.234084 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.234132 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.234204 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20250903 14:27:31.235133 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.231506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44633 (local address 127.19.65.130:33293)
0903 14:27:31.231742 (+ 236us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.231746 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.231756 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:31.231785 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.231787 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.231824 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.231884 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.232314 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.232955 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.234461 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.234903 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.234937 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.234984 (+ 47us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.235036 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20250903 14:27:31.233461 28408 leader_election.cc:290] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.235945 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74cfb5f34323411a82426ee83f90a13e" candidate_uuid: "f356a5927e614fbab65e145d538b27e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9" is_pre_election: true
I20250903 14:27:31.236166 28178 raft_consensus.cc:2466] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f356a5927e614fbab65e145d538b27e8 in term 0.
I20250903 14:27:31.236546 27978 leader_election.cc:304] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa25bdfbcba426a8348dcfce02037d9, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.236824 28408 raft_consensus.cc:2802] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:31.236927 28408 raft_consensus.cc:491] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:31.237008 28408 raft_consensus.cc:3058] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.237612 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.233135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42063 (local address 127.19.65.129:52733)
0903 14:27:31.233977 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.234000 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.234103 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.234501 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.234504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.234520 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.234762 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.234770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.235912 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.235915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.236681 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.236690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.237327 (+ 637us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.237340 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.237410 (+ 70us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.237459 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":77,"threads_started":1}
I20250903 14:27:31.237643 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.233405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:52733 (local address 127.19.65.131:42063)
0903 14:27:31.233529 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.233532 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.234050 (+ 518us) server_negotiation.cc:408] Connection header received
0903 14:27:31.234210 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.234213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.234250 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.234288 (+ 38us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.234913 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.235601 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.236817 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.237244 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.237276 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.237498 (+ 222us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.237559 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20250903 14:27:31.238114 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74cfb5f34323411a82426ee83f90a13e" candidate_uuid: "f356a5927e614fbab65e145d538b27e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f" is_pre_election: true
I20250903 14:27:31.238250 28408 raft_consensus.cc:513] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.238328 28326 raft_consensus.cc:2466] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f356a5927e614fbab65e145d538b27e8 in term 0.
I20250903 14:27:31.238739 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74cfb5f34323411a82426ee83f90a13e" candidate_uuid: "f356a5927e614fbab65e145d538b27e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9"
I20250903 14:27:31.238780 28408 leader_election.cc:290] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.238833 28178 raft_consensus.cc:3058] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.239091 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74cfb5f34323411a82426ee83f90a13e" candidate_uuid: "f356a5927e614fbab65e145d538b27e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f"
I20250903 14:27:31.239181 28326 raft_consensus.cc:3058] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.239653 28178 raft_consensus.cc:2466] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f356a5927e614fbab65e145d538b27e8 in term 1.
I20250903 14:27:31.239884 27978 leader_election.cc:304] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa25bdfbcba426a8348dcfce02037d9, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.239929 28326 raft_consensus.cc:2466] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f356a5927e614fbab65e145d538b27e8 in term 1.
I20250903 14:27:31.240043 28408 raft_consensus.cc:2802] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:31.240170 28408 raft_consensus.cc:695] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 1 LEADER]: Becoming Leader. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Running, Role: LEADER
I20250903 14:27:31.240386 28408 consensus_queue.cc:237] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.242668 27885 catalog_manager.cc:5582] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f356a5927e614fbab65e145d538b27e8 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "f356a5927e614fbab65e145d538b27e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } health_report { overall_health: UNKNOWN } } }
I20250903 14:27:31.250936 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.248718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.19.65.129:42367)
0903 14:27:31.248803 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.248806 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.248816 (+ 10us) server_negotiation.cc:408] Connection header received
0903 14:27:31.248923 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.248925 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.248973 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.249039 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.249277 (+ 238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.249675 (+ 398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.250189 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.250297 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.250335 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.250658 (+ 323us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.250734 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.250790 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.250824 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20250903 14:27:31.255120 28178 raft_consensus.cc:1273] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Refusing update from remote peer f356a5927e614fbab65e145d538b27e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.255158 28326 raft_consensus.cc:1273] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Refusing update from remote peer f356a5927e614fbab65e145d538b27e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.255637 28408 consensus_queue.cc:1035] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.255765 28410 consensus_queue.cc:1035] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.262926 28417 mvcc.cc:204] Tried to move back new op lower bound from 7196301931536007168 to 7196301931482361856. Current Snapshot: MvccSnapshot[applied={T|T < 7196301931536007168}]
I20250903 14:27:31.268734 27885 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.268832 27885 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.270653 27885 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Sending DeleteTablet for 3 replicas of tablet 74cfb5f34323411a82426ee83f90a13e
I20250903 14:27:31.271282 28166 tablet_service.cc:1515] Processing DeleteTablet for tablet 74cfb5f34323411a82426ee83f90a13e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34468
I20250903 14:27:31.271389 28308 tablet_service.cc:1515] Processing DeleteTablet for tablet 74cfb5f34323411a82426ee83f90a13e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59014
I20250903 14:27:31.271641 28024 tablet_service.cc:1515] Processing DeleteTablet for tablet 74cfb5f34323411a82426ee83f90a13e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46758
I20250903 14:27:31.271871 28424 tablet_replica.cc:331] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: stopping tablet replica
I20250903 14:27:31.271970 28426 tablet_replica.cc:331] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: stopping tablet replica
I20250903 14:27:31.271903 27885 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:27:31.272156 28426 raft_consensus.cc:2241] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 1 LEADER]: Raft consensus shutting down.
W20250903 14:27:31.272243 27885 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:31.272336 28424 raft_consensus.cc:2241] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.272480 28426 raft_consensus.cc:2270] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.272657 28424 raft_consensus.cc:2270] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.273772 28426 ts_tablet_manager.cc:1905] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.274654 28424 ts_tablet_manager.cc:1905] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.276384 28024 tablet_service.cc:1468] Processing CreateTablet for tablet b1e18de4d9474ee88520ad54080cacbe (DEFAULT_TABLE table=test-table [id=59c39c31992e476491704f97e37d3831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.276599 28024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e18de4d9474ee88520ad54080cacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.277136 28424 ts_tablet_manager.cc:1918] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.277215 28424 log.cc:1199] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/74cfb5f34323411a82426ee83f90a13e
I20250903 14:27:31.277505 28424 ts_tablet_manager.cc:1939] T 74cfb5f34323411a82426ee83f90a13e P 3aa25bdfbcba426a8348dcfce02037d9: Deleting consensus metadata
I20250903 14:27:31.277949 28308 tablet_service.cc:1468] Processing CreateTablet for tablet b1e18de4d9474ee88520ad54080cacbe (DEFAULT_TABLE table=test-table [id=59c39c31992e476491704f97e37d3831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.278159 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e18de4d9474ee88520ad54080cacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.278365 28166 tablet_service.cc:1468] Processing CreateTablet for tablet b1e18de4d9474ee88520ad54080cacbe (DEFAULT_TABLE table=test-table [id=59c39c31992e476491704f97e37d3831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.278429 27872 catalog_manager.cc:4928] TS 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293): tablet 74cfb5f34323411a82426ee83f90a13e (table test-table [id=3a31e345a3c0416c83153457c6622b30]) successfully deleted
I20250903 14:27:31.278551 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e18de4d9474ee88520ad54080cacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.280092 28403 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Bootstrap starting.
I20250903 14:27:31.280320 28404 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Bootstrap starting.
I20250903 14:27:31.280987 28403 tablet_bootstrap.cc:654] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.281199 28404 tablet_bootstrap.cc:654] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.281589 28402 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Bootstrap starting.
I20250903 14:27:31.282383 28403 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: No bootstrap required, opened a new log
I20250903 14:27:31.282439 28402 tablet_bootstrap.cc:654] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.282460 28403 ts_tablet_manager.cc:1397] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:31.282640 28425 tablet_replica.cc:331] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: stopping tablet replica
I20250903 14:27:31.282811 28403 raft_consensus.cc:357] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.282902 28403 raft_consensus.cc:383] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.282928 28403 raft_consensus.cc:738] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.283030 28403 consensus_queue.cc:260] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.283214 28403 ts_tablet_manager.cc:1428] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:31.283501 28425 raft_consensus.cc:2241] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.283780 28425 raft_consensus.cc:2270] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.284178 28426 ts_tablet_manager.cc:1918] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.284482 28402 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: No bootstrap required, opened a new log
I20250903 14:27:31.284554 28402 ts_tablet_manager.cc:1397] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:27:31.284993 28402 raft_consensus.cc:357] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.285094 28402 raft_consensus.cc:383] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.285125 28402 raft_consensus.cc:738] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.285145 28425 ts_tablet_manager.cc:1905] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.285246 28402 consensus_queue.cc:260] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.285490 28402 ts_tablet_manager.cc:1428] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:31.287163 28404 tablet_bootstrap.cc:492] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: No bootstrap required, opened a new log
I20250903 14:27:31.287292 28404 ts_tablet_manager.cc:1397] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250903 14:27:31.287310 28425 ts_tablet_manager.cc:1918] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.287382 28425 log.cc:1199] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/74cfb5f34323411a82426ee83f90a13e
I20250903 14:27:31.287657 28425 ts_tablet_manager.cc:1939] T 74cfb5f34323411a82426ee83f90a13e P 24c2eb0cfcbe45d29515257222ee617f: Deleting consensus metadata
I20250903 14:27:31.287663 28404 raft_consensus.cc:357] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.287765 28404 raft_consensus.cc:383] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.287803 28404 raft_consensus.cc:738] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.287896 28404 consensus_queue.cc:260] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.288111 28404 ts_tablet_manager.cc:1428] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:31.288441 28426 log.cc:1199] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/74cfb5f34323411a82426ee83f90a13e
I20250903 14:27:31.288709 28426 ts_tablet_manager.cc:1939] T 74cfb5f34323411a82426ee83f90a13e P f356a5927e614fbab65e145d538b27e8: Deleting consensus metadata
I20250903 14:27:31.289623 27870 catalog_manager.cc:4928] TS f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367): tablet 74cfb5f34323411a82426ee83f90a13e (table test-table [id=3a31e345a3c0416c83153457c6622b30]) successfully deleted
I20250903 14:27:31.290341 27872 catalog_manager.cc:4928] TS 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063): tablet 74cfb5f34323411a82426ee83f90a13e (table test-table [id=3a31e345a3c0416c83153457c6622b30]) successfully deleted
I20250903 14:27:31.442066 28409 raft_consensus.cc:491] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:31.442193 28409 raft_consensus.cc:513] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.443128 28409 leader_election.cc:290] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.445744 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.443304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42367 (local address 127.19.65.130:50601)
0903 14:27:31.443411 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.443426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.443508 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.443768 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.443773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.443797 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.444014 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.444023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.444712 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.444715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.445362 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.445371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.445459 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.445510 (+ 51us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.445558 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.445621 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 14:27:31.446103 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.443347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42063 (local address 127.19.65.130:56411)
0903 14:27:31.443677 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.443689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.443765 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.443982 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.443984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.443995 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.444228 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.444245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.444898 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.444901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.445515 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.445524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.445629 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.445647 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.445695 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.445749 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":82,"threads_started":1}
I20250903 14:27:31.446228 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.443381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50601 (local address 127.19.65.129:42367)
0903 14:27:31.443494 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.443498 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.443512 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:31.443629 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.443634 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.443682 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.443750 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.444131 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.444608 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.445510 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.445977 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.446035 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.446078 (+ 43us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.446132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:31.446336 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.443605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:56411 (local address 127.19.65.131:42063)
0903 14:27:31.443744 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.443749 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.443760 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:31.443857 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.443860 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.443902 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.443969 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.444341 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.444792 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.445627 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.446096 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.446122 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.446167 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.446225 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 14:27:31.446630 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e18de4d9474ee88520ad54080cacbe" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8" is_pre_election: true
I20250903 14:27:31.446635 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e18de4d9474ee88520ad54080cacbe" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f" is_pre_election: true
I20250903 14:27:31.446740 28326 raft_consensus.cc:2466] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 0.
I20250903 14:27:31.446791 28044 raft_consensus.cc:2466] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 0.
I20250903 14:27:31.447007 28120 leader_election.cc:304] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, 3aa25bdfbcba426a8348dcfce02037d9; no voters:
I20250903 14:27:31.447254 28409 raft_consensus.cc:2802] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:31.447309 28409 raft_consensus.cc:491] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:31.447355 28409 raft_consensus.cc:3058] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.448221 28409 raft_consensus.cc:513] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.448524 28409 leader_election.cc:290] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 election: Requested vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.448829 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e18de4d9474ee88520ad54080cacbe" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8"
I20250903 14:27:31.448818 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e18de4d9474ee88520ad54080cacbe" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f"
I20250903 14:27:31.448927 28326 raft_consensus.cc:3058] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.448926 28044 raft_consensus.cc:3058] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.449909 28044 raft_consensus.cc:2466] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 1.
I20250903 14:27:31.449909 28326 raft_consensus.cc:2466] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 1.
I20250903 14:27:31.450153 28118 leader_election.cc:304] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa25bdfbcba426a8348dcfce02037d9, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.450294 28409 raft_consensus.cc:2802] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:31.450412 28409 raft_consensus.cc:695] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Running, Role: LEADER
I20250903 14:27:31.450591 28409 consensus_queue.cc:237] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.452417 27884 catalog_manager.cc:5582] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "3aa25bdfbcba426a8348dcfce02037d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } health_report { overall_health: UNKNOWN } } }
W20250903 14:27:31.476894 28233 tablet.cc:2378] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909650, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:31.489511 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.482887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:34745 (local address 127.0.0.1:58618)
0903 14:27:31.483115 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.483126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.483209 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.483393 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.483395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.483769 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:31.483936 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.483941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.484502 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.484503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.484921 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.484928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.485026 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.485588 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:31.485601 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:31.486934 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:31.488446 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.488453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.488457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.488638 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.488819 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:31.488820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:31.488822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:31.488857 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:31.489091 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:31.489095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:31.489176 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.489257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.489362 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20250903 14:27:31.489609 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.482951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.19.65.190:34745)
0903 14:27:31.483077 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.483080 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.483187 (+ 107us) server_negotiation.cc:408] Connection header received
0903 14:27:31.483274 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.483277 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.483312 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.483371 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.484039 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.484433 (+ 394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.485016 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.485140 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.487067 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:31.487080 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:31.487083 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:31.487102 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:31.488354 (+ 1252us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.488716 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.488718 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.488720 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.488755 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.488909 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.488911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.488912 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.489027 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:31.489088 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.489295 (+ 207us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.489426 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":57}
I20250903 14:27:31.492658 27884 catalog_manager.cc:2232] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:58618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 14:27:31.493351 27884 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:31.497530 28024 tablet_service.cc:1468] Processing CreateTablet for tablet c089faf089584a46835e997eae683704 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4f1cb0367194b7f8afa88c643afd61b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:27:31.497587 28166 tablet_service.cc:1468] Processing CreateTablet for tablet c089faf089584a46835e997eae683704 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4f1cb0367194b7f8afa88c643afd61b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:27:31.497867 28024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c089faf089584a46835e997eae683704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.497910 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c089faf089584a46835e997eae683704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.499876 28404 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: Bootstrap starting.
I20250903 14:27:31.499944 28402 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: Bootstrap starting.
I20250903 14:27:31.500289 28308 tablet_service.cc:1468] Processing CreateTablet for tablet c089faf089584a46835e997eae683704 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4f1cb0367194b7f8afa88c643afd61b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 14:27:31.500581 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c089faf089584a46835e997eae683704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.500682 28404 tablet_bootstrap.cc:654] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.500701 28402 tablet_bootstrap.cc:654] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.501718 28402 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: No bootstrap required, opened a new log
I20250903 14:27:31.501819 28402 ts_tablet_manager.cc:1397] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:31.502288 28404 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: No bootstrap required, opened a new log
I20250903 14:27:31.502261 28402 raft_consensus.cc:357] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.502353 28402 raft_consensus.cc:383] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.502368 28404 ts_tablet_manager.cc:1397] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250903 14:27:31.502390 28402 raft_consensus.cc:738] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.502518 28402 consensus_queue.cc:260] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.502717 28404 raft_consensus.cc:357] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.502759 28402 ts_tablet_manager.cc:1428] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:31.502797 28404 raft_consensus.cc:383] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.502835 28404 raft_consensus.cc:738] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.502954 28409 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.502954 28404 consensus_queue.cc:260] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.503063 28409 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.503206 28404 ts_tablet_manager.cc:1428] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:31.503189 28410 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.503278 28410 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.503547 28403 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: Bootstrap starting.
I20250903 14:27:31.504276 28403 tablet_bootstrap.cc:654] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.505322 28403 tablet_bootstrap.cc:492] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: No bootstrap required, opened a new log
I20250903 14:27:31.505400 28403 ts_tablet_manager.cc:1397] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250903 14:27:31.505843 28403 raft_consensus.cc:357] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.505931 28403 raft_consensus.cc:383] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.505967 28403 raft_consensus.cc:738] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.506099 28403 consensus_queue.cc:260] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.506318 28403 ts_tablet_manager.cc:1428] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:31.506314 28411 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.506448 28411 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.519301 28411 raft_consensus.cc:491] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:31.519367 28411 raft_consensus.cc:513] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.520179 28411 leader_election.cc:290] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293)
I20250903 14:27:31.522738 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.520338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42367 (local address 127.19.65.131:59463)
0903 14:27:31.520443 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.520459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.520529 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.520794 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.520797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.520815 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.521042 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.521050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.521877 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.521879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.522386 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.522392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.522492 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.522510 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.522556 (+ 46us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.522617 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 14:27:31.523162 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.520410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59463 (local address 127.19.65.129:42367)
0903 14:27:31.520532 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.520537 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.520551 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:31.520605 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.520608 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.520676 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.520742 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.521157 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.521798 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.522497 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.522978 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.523008 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.523045 (+ 37us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.523085 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:31.523468 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c089faf089584a46835e997eae683704" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8" is_pre_election: true
I20250903 14:27:31.523586 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.520412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46387 (local address 127.19.65.130:33293)
0903 14:27:31.520504 (+ 92us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.520507 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.520894 (+ 387us) server_negotiation.cc:408] Connection header received
0903 14:27:31.521044 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.521046 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.521080 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.521140 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:31.521600 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.522098 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.522890 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.523189 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.523224 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.523436 (+ 212us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.523497 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250903 14:27:31.523581 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.520338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33293 (local address 127.19.65.131:46387)
0903 14:27:31.520821 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.520836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.520947 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.521184 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.521190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.521217 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:31.521486 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.521517 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.522192 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.522195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.522756 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.522763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.523314 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.523329 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.523400 (+ 71us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.523451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":250,"thread_start_us":69,"threads_started":1}
I20250903 14:27:31.523571 28044 raft_consensus.cc:2466] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 0.
I20250903 14:27:31.523905 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c089faf089584a46835e997eae683704" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9" is_pre_election: true
I20250903 14:27:31.523905 28260 leader_election.cc:304] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.524004 28178 raft_consensus.cc:2466] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 0.
I20250903 14:27:31.524111 28411 raft_consensus.cc:2802] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:31.524178 28411 raft_consensus.cc:491] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:31.524235 28411 raft_consensus.cc:3058] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.524945 28411 raft_consensus.cc:513] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.525279 28411 leader_election.cc:290] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 election: Requested vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293)
I20250903 14:27:31.525554 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c089faf089584a46835e997eae683704" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8"
I20250903 14:27:31.525650 28044 raft_consensus.cc:3058] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.525635 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c089faf089584a46835e997eae683704" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9"
I20250903 14:27:31.525730 28178 raft_consensus.cc:3058] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.526398 28044 raft_consensus.cc:2466] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 1.
I20250903 14:27:31.526468 28178 raft_consensus.cc:2466] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 1.
I20250903 14:27:31.526635 28260 leader_election.cc:304] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.526777 28411 raft_consensus.cc:2802] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:31.526952 28411 raft_consensus.cc:695] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [term 1 LEADER]: Becoming Leader. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Running, Role: LEADER
I20250903 14:27:31.527137 28411 consensus_queue.cc:237] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.528383 28443 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: TxnStatusTablet state changed. Reason: New leader 24c2eb0cfcbe45d29515257222ee617f. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.528471 28443 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:27:31.528707 28445 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 14:27:31.528824 28445 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 14:27:31.529198 27885 catalog_manager.cc:5582] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131). New cstate: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } health_report { overall_health: HEALTHY } } }
W20250903 14:27:31.533550 28091 tablet.cc:2378] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:31.544101 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.541008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.19.65.131:42063)
0903 14:27:31.541103 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.541108 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.541191 (+ 83us) server_negotiation.cc:408] Connection header received
0903 14:27:31.541368 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.541372 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.541421 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.541519 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.541837 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.542426 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.543182 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.543344 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.543393 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.543800 (+ 407us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.543895 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.543967 (+ 72us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.544005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 14:27:31.544111 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.540939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42063 (local address 127.0.0.1:59022)
0903 14:27:31.541126 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.541137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.541246 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.541541 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.541544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.541552 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:27:31.541741 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.541746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.542541 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.542545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.543075 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.543083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.543185 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.543233 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.543884 (+ 651us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.543894 (+ 10us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.543950 (+ 56us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.544011 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20250903 14:27:31.546864 28178 raft_consensus.cc:1273] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Refusing update from remote peer 24c2eb0cfcbe45d29515257222ee617f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.547094 28044 raft_consensus.cc:1273] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Refusing update from remote peer 24c2eb0cfcbe45d29515257222ee617f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.547266 28411 consensus_queue.cc:1035] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.547799 28411 consensus_queue.cc:1035] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.548532 28409 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: TxnStatusTablet state changed. Reason: New leader 24c2eb0cfcbe45d29515257222ee617f. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.548630 28409 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.549139 28410 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: TxnStatusTablet state changed. Reason: New leader 24c2eb0cfcbe45d29515257222ee617f. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.549225 28410 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.550019 28411 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.550112 28411 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:27:31.551949 28409 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.552042 28409 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 3aa25bdfbcba426a8348dcfce02037d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.552130 28416 mvcc.cc:204] Tried to move back new op lower bound from 7196301932732358656 to 7196301932657991680. Current Snapshot: MvccSnapshot[applied={T|T < 7196301932732358656 or (T in {7196301932732358656})}]
I20250903 14:27:31.555630 28443 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.555716 28443 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P 24c2eb0cfcbe45d29515257222ee617f: This TxnStatusTablet replica's current role is: LEADER
I20250903 14:27:31.556355 28415 mvcc.cc:204] Tried to move back new op lower bound from 7196301932732358656 to 7196301932657991680. Current Snapshot: MvccSnapshot[applied={T|T < 7196301932732358656}]
I20250903 14:27:31.556514 28410 tablet_replica.cc:440] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } } }
I20250903 14:27:31.556602 28410 tablet_replica.cc:443] T c089faf089584a46835e997eae683704 P f356a5927e614fbab65e145d538b27e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 14:27:31.557596 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.553839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.19.65.130:33293)
0903 14:27:31.553946 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.553950 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.553961 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:31.554018 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.554020 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.554065 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.554137 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.554575 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.555239 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.556220 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.556384 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.556445 (+ 61us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.556900 (+ 455us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.557000 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.557432 (+ 432us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.557490 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 14:27:31.568027 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.563292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42063 (local address 127.19.65.130:55783)
0903 14:27:31.565680 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.565692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.565775 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.566024 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.566026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.566031 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:27:31.566199 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.566203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.567016 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.567018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.567390 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.567396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.567471 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.567496 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.567833 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.567836 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.567875 (+ 39us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.567919 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2307,"thread_start_us":101,"threads_started":1}
I20250903 14:27:31.568049 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.563354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:55783 (local address 127.19.65.131:42063)
0903 14:27:31.563444 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.563448 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.565765 (+ 2317us) server_negotiation.cc:408] Connection header received
0903 14:27:31.565855 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.565859 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.565906 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.565995 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.566299 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.566923 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.567493 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.567657 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.567694 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.567758 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.567827 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.567896 (+ 69us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.567933 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 14:27:31.573768 28326 raft_consensus.cc:1273] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Refusing update from remote peer 3aa25bdfbcba426a8348dcfce02037d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.573884 28044 raft_consensus.cc:1273] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Refusing update from remote peer 3aa25bdfbcba426a8348dcfce02037d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.574287 28431 consensus_queue.cc:1035] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.574414 28409 consensus_queue.cc:1035] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.596864 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.594133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33293 (local address 127.19.65.131:56273)
0903 14:27:31.594334 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:31.594346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:31.594423 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:31.594677 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:31.594680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:31.594689 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 14:27:31.594941 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.594948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.595886 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.595889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:31.596292 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:31.596299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.596374 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.596402 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.596704 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.596707 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 14:27:31.596745 (+ 38us) client_negotiation.cc:240] Negotiation successful
0903 14:27:31.596787 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20250903 14:27:31.596896 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.594185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:56273 (local address 127.19.65.130:33293)
0903 14:27:31.594291 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.594295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.594412 (+ 117us) server_negotiation.cc:408] Connection header received
0903 14:27:31.594493 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.594495 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.594541 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.594603 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.595063 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.595773 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.596379 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.596537 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.596573 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.596634 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.596698 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.596766 (+ 68us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.596802 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250903 14:27:31.619724 27884 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.619849 27884 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.621754 27884 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Sending DeleteTablet for 3 replicas of tablet b1e18de4d9474ee88520ad54080cacbe
I20250903 14:27:31.622159 28024 tablet_service.cc:1515] Processing DeleteTablet for tablet b1e18de4d9474ee88520ad54080cacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46758
I20250903 14:27:31.622449 28426 tablet_replica.cc:331] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: stopping tablet replica
I20250903 14:27:31.622558 28426 raft_consensus.cc:2241] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.622656 28426 raft_consensus.cc:2270] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.622790 28166 tablet_service.cc:1515] Processing DeleteTablet for tablet b1e18de4d9474ee88520ad54080cacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34468
I20250903 14:27:31.622926 28308 tablet_service.cc:1515] Processing DeleteTablet for tablet b1e18de4d9474ee88520ad54080cacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59014
I20250903 14:27:31.623060 28424 tablet_replica.cc:331] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: stopping tablet replica
I20250903 14:27:31.623070 28425 tablet_replica.cc:331] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: stopping tablet replica
I20250903 14:27:31.623142 28424 raft_consensus.cc:2241] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:31.623168 28425 raft_consensus.cc:2241] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.623260 28425 raft_consensus.cc:2270] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.623351 28424 raft_consensus.cc:2270] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.623955 28426 ts_tablet_manager.cc:1905] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.623955 28425 ts_tablet_manager.cc:1905] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.626364 28426 ts_tablet_manager.cc:1918] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:27:31.626370 28425 ts_tablet_manager.cc:1918] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:27:31.626416 28426 log.cc:1199] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/b1e18de4d9474ee88520ad54080cacbe
I20250903 14:27:31.626426 28425 log.cc:1199] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/b1e18de4d9474ee88520ad54080cacbe
I20250903 14:27:31.626631 28426 ts_tablet_manager.cc:1939] T b1e18de4d9474ee88520ad54080cacbe P f356a5927e614fbab65e145d538b27e8: Deleting consensus metadata
I20250903 14:27:31.626663 28425 ts_tablet_manager.cc:1939] T b1e18de4d9474ee88520ad54080cacbe P 24c2eb0cfcbe45d29515257222ee617f: Deleting consensus metadata
I20250903 14:27:31.627183 28424 ts_tablet_manager.cc:1905] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.627182 27872 catalog_manager.cc:4928] TS 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063): tablet b1e18de4d9474ee88520ad54080cacbe (table test-table [id=59c39c31992e476491704f97e37d3831]) successfully deleted
I20250903 14:27:31.629227 28424 ts_tablet_manager.cc:1918] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:27:31.629300 28424 log.cc:1199] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/b1e18de4d9474ee88520ad54080cacbe
I20250903 14:27:31.629518 28424 ts_tablet_manager.cc:1939] T b1e18de4d9474ee88520ad54080cacbe P 3aa25bdfbcba426a8348dcfce02037d9: Deleting consensus metadata
I20250903 14:27:31.629631 27870 catalog_manager.cc:4928] TS f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367): tablet b1e18de4d9474ee88520ad54080cacbe (table test-table [id=59c39c31992e476491704f97e37d3831]) successfully deleted
I20250903 14:27:31.630079 27872 catalog_manager.cc:4928] TS 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293): tablet b1e18de4d9474ee88520ad54080cacbe (table test-table [id=59c39c31992e476491704f97e37d3831]) successfully deleted
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20250903 14:27:31.655226 27935 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:31.649225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.19.65.190:34745)
0903 14:27:31.649333 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.649337 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.649568 (+ 231us) server_negotiation.cc:408] Connection header received
0903 14:27:31.649685 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.649688 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.649733 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.649804 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.650658 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.651325 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.652077 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.652595 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.655120 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55086: BlockingRecv error: recv got EOF from 127.0.0.1:55086 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 03 14:27:31 dist-test-slave-4wpb krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20250903 14:27:31.664728 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.656517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.19.65.190:34745)
0903 14:27:31.656604 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.656607 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.656956 (+ 349us) server_negotiation.cc:408] Connection header received
0903 14:27:31.657110 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.657113 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.657155 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.657231 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:31.658067 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.658746 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.659533 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.660047 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.661847 (+ 1800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:31.661863 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:31.661866 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:31.661889 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:31.663452 (+ 1563us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.663819 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.663822 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.663824 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.663865 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:31.664064 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:31.664067 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:31.664068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:31.664205 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:31.664321 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.664497 (+ 176us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.664599 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 14:27:31.667145 27884 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:31.667557 27884 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:31.672142 28166 tablet_service.cc:1468] Processing CreateTablet for tablet e4b91924c92d4498aba26d383af911b5 (DEFAULT_TABLE table=test-table [id=64f145bed79842fb910666c07dea1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.672142 28308 tablet_service.cc:1468] Processing CreateTablet for tablet e4b91924c92d4498aba26d383af911b5 (DEFAULT_TABLE table=test-table [id=64f145bed79842fb910666c07dea1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.672389 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b91924c92d4498aba26d383af911b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.672389 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b91924c92d4498aba26d383af911b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.673705 28024 tablet_service.cc:1468] Processing CreateTablet for tablet e4b91924c92d4498aba26d383af911b5 (DEFAULT_TABLE table=test-table [id=64f145bed79842fb910666c07dea1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.673933 28024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b91924c92d4498aba26d383af911b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.674458 28403 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Bootstrap starting.
I20250903 14:27:31.675370 28403 tablet_bootstrap.cc:654] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.675992 28404 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Bootstrap starting.
I20250903 14:27:31.676693 28404 tablet_bootstrap.cc:654] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.677819 28404 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: No bootstrap required, opened a new log
I20250903 14:27:31.677910 28404 ts_tablet_manager.cc:1397] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:31.678267 28404 raft_consensus.cc:357] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.678361 28404 raft_consensus.cc:383] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.678388 28404 raft_consensus.cc:738] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.678503 28404 consensus_queue.cc:260] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.678716 28404 ts_tablet_manager.cc:1428] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:31.680153 28402 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Bootstrap starting.
I20250903 14:27:31.681007 28402 tablet_bootstrap.cc:654] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.682096 28402 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: No bootstrap required, opened a new log
I20250903 14:27:31.682170 28402 ts_tablet_manager.cc:1397] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:31.682556 28402 raft_consensus.cc:357] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.682658 28402 raft_consensus.cc:383] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.682689 28402 raft_consensus.cc:738] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.682796 28402 consensus_queue.cc:260] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.683013 28402 ts_tablet_manager.cc:1428] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:31.686193 28403 tablet_bootstrap.cc:492] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: No bootstrap required, opened a new log
I20250903 14:27:31.686291 28403 ts_tablet_manager.cc:1397] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250903 14:27:31.686725 28403 raft_consensus.cc:357] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.686816 28403 raft_consensus.cc:383] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.686846 28403 raft_consensus.cc:738] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.686956 28403 consensus_queue.cc:260] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.687183 28403 ts_tablet_manager.cc:1428] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 14:27:31.853438 28443 raft_consensus.cc:491] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:31.853610 28443 raft_consensus.cc:513] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.854050 28443 leader_election.cc:290] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293)
I20250903 14:27:31.854437 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b91924c92d4498aba26d383af911b5" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9" is_pre_election: true
I20250903 14:27:31.854423 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b91924c92d4498aba26d383af911b5" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8" is_pre_election: true
I20250903 14:27:31.854569 28178 raft_consensus.cc:2466] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 0.
I20250903 14:27:31.854566 28044 raft_consensus.cc:2466] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 0.
I20250903 14:27:31.854853 28262 leader_election.cc:304] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, 3aa25bdfbcba426a8348dcfce02037d9; no voters:
I20250903 14:27:31.855028 28443 raft_consensus.cc:2802] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:31.855086 28443 raft_consensus.cc:491] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:31.855141 28443 raft_consensus.cc:3058] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.856501 28443 raft_consensus.cc:513] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.856853 28443 leader_election.cc:290] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 election: Requested vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293)
I20250903 14:27:31.857197 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b91924c92d4498aba26d383af911b5" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8"
I20250903 14:27:31.857208 28178 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b91924c92d4498aba26d383af911b5" candidate_uuid: "24c2eb0cfcbe45d29515257222ee617f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa25bdfbcba426a8348dcfce02037d9"
I20250903 14:27:31.857296 28178 raft_consensus.cc:3058] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.857309 28044 raft_consensus.cc:3058] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.858058 28178 raft_consensus.cc:2466] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 1.
I20250903 14:27:31.858084 28044 raft_consensus.cc:2466] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c2eb0cfcbe45d29515257222ee617f in term 1.
I20250903 14:27:31.858318 28262 leader_election.cc:304] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, 3aa25bdfbcba426a8348dcfce02037d9; no voters:
I20250903 14:27:31.858477 28443 raft_consensus.cc:2802] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:31.858552 28443 raft_consensus.cc:695] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 1 LEADER]: Becoming Leader. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Running, Role: LEADER
I20250903 14:27:31.858678 28443 consensus_queue.cc:237] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.860021 27885 catalog_manager.cc:5582] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131). New cstate: current_term: 1 leader_uuid: "24c2eb0cfcbe45d29515257222ee617f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } health_report { overall_health: HEALTHY } } }
I20250903 14:27:31.869608 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.867464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.19.65.131:42063)
0903 14:27:31.867584 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.867588 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.867599 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:31.867700 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.867703 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.867744 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.867801 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.868082 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.868478 (+ 396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.869022 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.869185 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.869221 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.869278 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.869336 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.869435 (+ 99us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.869494 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 14:27:31.871305 28044 raft_consensus.cc:1273] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Refusing update from remote peer 24c2eb0cfcbe45d29515257222ee617f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.871315 28178 raft_consensus.cc:1273] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Refusing update from remote peer 24c2eb0cfcbe45d29515257222ee617f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.871692 28411 consensus_queue.cc:1035] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.871800 28443 consensus_queue.cc:1035] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.880095 27883 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.880183 27883 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.881706 27883 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Sending DeleteTablet for 3 replicas of tablet e4b91924c92d4498aba26d383af911b5
I20250903 14:27:31.882115 28024 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b91924c92d4498aba26d383af911b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46758
I20250903 14:27:31.882241 28308 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b91924c92d4498aba26d383af911b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59014
I20250903 14:27:31.882391 28425 tablet_replica.cc:331] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: stopping tablet replica
I20250903 14:27:31.882400 28426 tablet_replica.cc:331] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: stopping tablet replica
I20250903 14:27:31.882500 28426 raft_consensus.cc:2241] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.882500 28425 raft_consensus.cc:2241] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:31.882601 28426 raft_consensus.cc:2270] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.882731 28425 raft_consensus.cc:2270] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.882841 28166 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b91924c92d4498aba26d383af911b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34468
I20250903 14:27:31.882973 28424 tablet_replica.cc:331] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: stopping tablet replica
I20250903 14:27:31.883057 28424 raft_consensus.cc:2241] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.883157 28424 raft_consensus.cc:2270] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.883347 28426 ts_tablet_manager.cc:1905] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.883471 28425 ts_tablet_manager.cc:1905] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.883996 28424 ts_tablet_manager.cc:1905] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.885427 28426 ts_tablet_manager.cc:1918] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.885514 28426 log.cc:1199] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-0/wal/wals/e4b91924c92d4498aba26d383af911b5
I20250903 14:27:31.885514 28425 ts_tablet_manager.cc:1918] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.885574 28425 log.cc:1199] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/e4b91924c92d4498aba26d383af911b5
I20250903 14:27:31.885784 28426 ts_tablet_manager.cc:1939] T e4b91924c92d4498aba26d383af911b5 P f356a5927e614fbab65e145d538b27e8: Deleting consensus metadata
I20250903 14:27:31.885819 28425 ts_tablet_manager.cc:1939] T e4b91924c92d4498aba26d383af911b5 P 24c2eb0cfcbe45d29515257222ee617f: Deleting consensus metadata
I20250903 14:27:31.886205 28424 ts_tablet_manager.cc:1918] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:31.886267 28424 log.cc:1199] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/e4b91924c92d4498aba26d383af911b5
I20250903 14:27:31.886384 27870 catalog_manager.cc:4928] TS f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367): tablet e4b91924c92d4498aba26d383af911b5 (table test-table [id=64f145bed79842fb910666c07dea1942]) successfully deleted
I20250903 14:27:31.886476 28424 ts_tablet_manager.cc:1939] T e4b91924c92d4498aba26d383af911b5 P 3aa25bdfbcba426a8348dcfce02037d9: Deleting consensus metadata
I20250903 14:27:31.886719 27872 catalog_manager.cc:4928] TS 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063): tablet e4b91924c92d4498aba26d383af911b5 (table test-table [id=64f145bed79842fb910666c07dea1942]) successfully deleted
I20250903 14:27:31.886795 27883 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:27:31.886976 27872 catalog_manager.cc:4928] TS 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293): tablet e4b91924c92d4498aba26d383af911b5 (table test-table [id=64f145bed79842fb910666c07dea1942]) successfully deleted
W20250903 14:27:31.887177 27883 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:31.890741 28024 tablet_service.cc:1468] Processing CreateTablet for tablet 925af723c2a345b3a5926bde8f21adaa (DEFAULT_TABLE table=test-table [id=9475d6788a4e48bbb16ba29b4a8082c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.890808 28308 tablet_service.cc:1468] Processing CreateTablet for tablet 925af723c2a345b3a5926bde8f21adaa (DEFAULT_TABLE table=test-table [id=9475d6788a4e48bbb16ba29b4a8082c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.890967 28024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925af723c2a345b3a5926bde8f21adaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.891007 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925af723c2a345b3a5926bde8f21adaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.892386 28404 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: Bootstrap starting.
I20250903 14:27:31.892526 28166 tablet_service.cc:1468] Processing CreateTablet for tablet 925af723c2a345b3a5926bde8f21adaa (DEFAULT_TABLE table=test-table [id=9475d6788a4e48bbb16ba29b4a8082c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:31.892671 28403 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Bootstrap starting.
I20250903 14:27:31.892704 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925af723c2a345b3a5926bde8f21adaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:31.893137 28404 tablet_bootstrap.cc:654] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.893401 28403 tablet_bootstrap.cc:654] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.894369 28403 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: No bootstrap required, opened a new log
I20250903 14:27:31.894426 28403 ts_tablet_manager.cc:1397] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:31.894572 28402 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Bootstrap starting.
I20250903 14:27:31.894739 28403 raft_consensus.cc:357] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.894876 28403 raft_consensus.cc:383] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.894912 28403 raft_consensus.cc:738] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c2eb0cfcbe45d29515257222ee617f, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.895031 28403 consensus_queue.cc:260] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.895265 28403 ts_tablet_manager.cc:1428] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 14:27:31.895386 28402 tablet_bootstrap.cc:654] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:31.895467 28404 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: No bootstrap required, opened a new log
I20250903 14:27:31.895551 28404 ts_tablet_manager.cc:1397] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250903 14:27:31.895927 28404 raft_consensus.cc:357] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.896064 28404 raft_consensus.cc:383] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.896107 28404 raft_consensus.cc:738] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a5927e614fbab65e145d538b27e8, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.896239 28404 consensus_queue.cc:260] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.896369 28402 tablet_bootstrap.cc:492] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: No bootstrap required, opened a new log
I20250903 14:27:31.896437 28402 ts_tablet_manager.cc:1397] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:31.896456 28404 ts_tablet_manager.cc:1428] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 14:27:31.896775 28402 raft_consensus.cc:357] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.896867 28402 raft_consensus.cc:383] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:31.896905 28402 raft_consensus.cc:738] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Initialized, Role: FOLLOWER
I20250903 14:27:31.897008 28402 consensus_queue.cc:260] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.897266 28402 ts_tablet_manager.cc:1428] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250903 14:27:31.918910 28375 tablet.cc:2378] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:31.937811 28456 raft_consensus.cc:491] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:31.937907 28456 raft_consensus.cc:513] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.938300 28456 leader_election.cc:290] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.938639 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "925af723c2a345b3a5926bde8f21adaa" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f" is_pre_election: true
I20250903 14:27:31.938639 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "925af723c2a345b3a5926bde8f21adaa" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8" is_pre_election: true
I20250903 14:27:31.938757 28044 raft_consensus.cc:2466] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 0.
I20250903 14:27:31.938757 28326 raft_consensus.cc:2466] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 0.
I20250903 14:27:31.939039 28118 leader_election.cc:304] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa25bdfbcba426a8348dcfce02037d9, f356a5927e614fbab65e145d538b27e8; no voters:
I20250903 14:27:31.939199 28456 raft_consensus.cc:2802] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:31.939265 28456 raft_consensus.cc:491] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:31.939316 28456 raft_consensus.cc:3058] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.940104 28456 raft_consensus.cc:513] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.940407 28456 leader_election.cc:290] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 election: Requested vote from peers f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367), 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063)
I20250903 14:27:31.940678 28326 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "925af723c2a345b3a5926bde8f21adaa" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c2eb0cfcbe45d29515257222ee617f"
I20250903 14:27:31.940711 28044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "925af723c2a345b3a5926bde8f21adaa" candidate_uuid: "3aa25bdfbcba426a8348dcfce02037d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f356a5927e614fbab65e145d538b27e8"
I20250903 14:27:31.940768 28326 raft_consensus.cc:3058] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.940824 28044 raft_consensus.cc:3058] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:31.941640 28326 raft_consensus.cc:2466] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 1.
I20250903 14:27:31.941854 28044 raft_consensus.cc:2466] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa25bdfbcba426a8348dcfce02037d9 in term 1.
I20250903 14:27:31.941871 28120 leader_election.cc:304] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c2eb0cfcbe45d29515257222ee617f, 3aa25bdfbcba426a8348dcfce02037d9; no voters:
I20250903 14:27:31.942049 28456 raft_consensus.cc:2802] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:31.942112 28456 raft_consensus.cc:695] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa25bdfbcba426a8348dcfce02037d9, State: Running, Role: LEADER
I20250903 14:27:31.942234 28456 consensus_queue.cc:237] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } }
I20250903 14:27:31.943589 27885 catalog_manager.cc:5582] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "3aa25bdfbcba426a8348dcfce02037d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa25bdfbcba426a8348dcfce02037d9" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 } health_report { overall_health: UNKNOWN } } }
I20250903 14:27:31.961216 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:31.958821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.19.65.130:33293)
0903 14:27:31.958927 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:31.958931 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:31.958944 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:31.959041 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:31.959044 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:31.959089 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:31.959164 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:31.959455 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.960135 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:31.960650 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:31.960815 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:31.960857 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:31.960923 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:31.960992 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:31.961063 (+ 71us) server_negotiation.cc:300] Negotiation successful
0903 14:27:31.961105 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:31.965879 28326 raft_consensus.cc:1273] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Refusing update from remote peer 3aa25bdfbcba426a8348dcfce02037d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.966229 28431 consensus_queue.cc:1035] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c2eb0cfcbe45d29515257222ee617f" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.966548 28044 raft_consensus.cc:1273] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Refusing update from remote peer 3aa25bdfbcba426a8348dcfce02037d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:31.967022 28431 consensus_queue.cc:1035] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f356a5927e614fbab65e145d538b27e8" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:31.993821 27885 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.993947 27885 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:31.995692 27885 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3f03fdc1b79e49ee86ac114b551af106: Sending DeleteTablet for 3 replicas of tablet 925af723c2a345b3a5926bde8f21adaa
I20250903 14:27:31.996138 28024 tablet_service.cc:1515] Processing DeleteTablet for tablet 925af723c2a345b3a5926bde8f21adaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:46758
I20250903 14:27:31.996302 28426 tablet_replica.cc:331] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: stopping tablet replica
I20250903 14:27:31.996405 28426 raft_consensus.cc:2241] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:31.996500 28426 raft_consensus.cc:2270] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:31.997393 28426 ts_tablet_manager.cc:1905] T 925af723c2a345b3a5926bde8f21adaa P f356a5927e614fbab65e145d538b27e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:31.997960 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27959
I20250903 14:27:31.999500 28166 tablet_service.cc:1515] Processing DeleteTablet for tablet 925af723c2a345b3a5926bde8f21adaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34468
I20250903 14:27:31.999647 28424 tablet_replica.cc:331] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: stopping tablet replica
I20250903 14:27:31.999756 28424 raft_consensus.cc:2241] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:31.999915 28424 raft_consensus.cc:2270] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:32.000241 28308 tablet_service.cc:1515] Processing DeleteTablet for tablet 925af723c2a345b3a5926bde8f21adaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:31 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59014
I20250903 14:27:32.000392 28425 tablet_replica.cc:331] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: stopping tablet replica
I20250903 14:27:32.000497 28425 raft_consensus.cc:2241] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:32.000579 28425 raft_consensus.cc:2270] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:32.000712 28424 ts_tablet_manager.cc:1905] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:32.001108 28425 ts_tablet_manager.cc:1905] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:32.002779 28424 ts_tablet_manager.cc:1918] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:27:32.002833 28424 log.cc:1199] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/925af723c2a345b3a5926bde8f21adaa
I20250903 14:27:32.003059 28424 ts_tablet_manager.cc:1939] T 925af723c2a345b3a5926bde8f21adaa P 3aa25bdfbcba426a8348dcfce02037d9: Deleting consensus metadata
I20250903 14:27:32.003159 28425 ts_tablet_manager.cc:1918] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 14:27:32.003209 28425 log.cc:1199] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/925af723c2a345b3a5926bde8f21adaa
I20250903 14:27:32.003405 28425 ts_tablet_manager.cc:1939] T 925af723c2a345b3a5926bde8f21adaa P 24c2eb0cfcbe45d29515257222ee617f: Deleting consensus metadata
I20250903 14:27:32.008845 27872 catalog_manager.cc:4928] TS 24c2eb0cfcbe45d29515257222ee617f (127.19.65.131:42063): tablet 925af723c2a345b3a5926bde8f21adaa (table test-table [id=9475d6788a4e48bbb16ba29b4a8082c7]) successfully deleted
I20250903 14:27:32.008983 27872 catalog_manager.cc:4928] TS 3aa25bdfbcba426a8348dcfce02037d9 (127.19.65.130:33293): tablet 925af723c2a345b3a5926bde8f21adaa (table test-table [id=9475d6788a4e48bbb16ba29b4a8082c7]) successfully deleted
W20250903 14:27:32.009430 27870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:42367 (error 108)
I20250903 14:27:32.010185 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28101
I20250903 14:27:32.010362 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:32.009999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42367 (local address 127.0.0.1:46526)
0903 14:27:32.010182 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:32.010254 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":47,"threads_started":1}
W20250903 14:27:32.011904 27870 catalog_manager.cc:4671] TS f356a5927e614fbab65e145d538b27e8 (127.19.65.129:42367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 925af723c2a345b3a5926bde8f21adaa: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42367: connect: Connection refused (error 111)
I20250903 14:27:32.017108 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28243
I20250903 14:27:32.022967 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 27853
2025-09-03T14:27:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3511 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:32 dist-test-slave-4wpb krb5kdc[28491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:32 dist-test-slave-4wpb krb5kdc[28491](info): set up 2 sockets
Sep 03 14:27:32 dist-test-slave-4wpb krb5kdc[28491](info): commencing operation
krb5kdc: starting...
W20250903 14:27:33.781939 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.738s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:33 dist-test-slave-4wpb krb5kdc[28491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:33.895738 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39633
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:33.984694 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:33.984861 28507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:33.984936 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:33.987716 28507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:33.987764 28507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:33.987797 28507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:33.987826 28507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:33.987857 28507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:33.991168 28507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:33.991896 28507 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:33.992595 28507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:33.996982 28515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:33.996975 28512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:33.996964 28513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:33.997306 28507 server_base.cc:1047] running on GCE node
I20250903 14:27:33.997706 28507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:33.998390 28507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:33.999517 28507 hybrid_clock.cc:648] HybridClock initialized: now 1756909653999508 us; error 26 us; skew 500 ppm
W20250903 14:27:34.001965 28507 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-09-03T14:27:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1978 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:34 dist-test-slave-4wpb krb5kdc[28520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:34 dist-test-slave-4wpb krb5kdc[28520](info): set up 2 sockets
Sep 03 14:27:34 dist-test-slave-4wpb krb5kdc[28520](info): commencing operation
krb5kdc: starting...
W20250903 14:27:35.793807 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.771s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:35 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:35.904110 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44265
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:35.993700 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:35.993870 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:35.993942 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:35.996789 28536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:35.996837 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:35.996868 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:35.996896 28536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:35.996928 28536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:36.000247 28536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.000993 28536 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.001663 28536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:36.006120 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.006114 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.006112 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.006422 28536 server_base.cc:1047] running on GCE node
I20250903 14:27:36.006847 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.007594 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.008755 28536 hybrid_clock.cc:648] HybridClock initialized: now 1756909656008757 us; error 33 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.011008 28536 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:27:36.011837 28536 webserver.cc:480] Webserver started at http://127.19.65.190:38355/ using document root <none> and password file <none>
I20250903 14:27:36.012271 28536 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.012317 28536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.012467 28536 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:36.013909 28536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "4651ea0480f54ca891efbf00adc090a5"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "c4f1618614f56e78df2f79bf6dd109a7"
server_key_iv: "bd768e956f2003b99e10e674e128675a"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.014322 28536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "4651ea0480f54ca891efbf00adc090a5"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "c4f1618614f56e78df2f79bf6dd109a7"
server_key_iv: "bd768e956f2003b99e10e674e128675a"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.017055 28536 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:27:36.018893 28551 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.019654 28536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:36.019742 28536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "4651ea0480f54ca891efbf00adc090a5"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "c4f1618614f56e78df2f79bf6dd109a7"
server_key_iv: "bd768e956f2003b99e10e674e128675a"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.019810 28536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:36.044045 28536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:36.046196 28536 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:36.046343 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:36.052209 28536 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44265
I20250903 14:27:36.052218 28603 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:44265 every 8 connection(s)
I20250903 14:27:36.052953 28536 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:36.054953 28604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:36.058514 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28536
I20250903 14:27:36.058619 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:36.058867 19718 external_mini_cluster.cc:1468] Setting key eedb4bac3edf4452f505539547fb238d
I20250903 14:27:36.059772 28604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Bootstrap starting.
I20250903 14:27:36.061657 28604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:36.062312 28604 log.cc:826] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:36.063925 28604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: No bootstrap required, opened a new log
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:36.066279 28604 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.066480 28604 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:36.066569 28604 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4651ea0480f54ca891efbf00adc090a5, State: Initialized, Role: FOLLOWER
I20250903 14:27:36.066951 28604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.067082 28604 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:36.067150 28604 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:36.067252 28604 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:36.068065 28604 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.068380 28604 leader_election.cc:304] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4651ea0480f54ca891efbf00adc090a5; no voters:
I20250903 14:27:36.068652 28604 leader_election.cc:290] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:36.068756 28609 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:36.068979 28609 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 LEADER]: Becoming Leader. State: Replica: 4651ea0480f54ca891efbf00adc090a5, State: Running, Role: LEADER
I20250903 14:27:36.069262 28609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.069518 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.059884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.19.65.190:44265)
0903 14:27:36.060187 (+ 303us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:36.060194 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:36.060221 (+ 27us) server_negotiation.cc:408] Connection header received
0903 14:27:36.060636 (+ 415us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:36.060648 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:36.060867 (+ 219us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:36.061077 (+ 210us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:36.062019 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.062751 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.063383 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.063622 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.065400 (+ 1778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:36.065419 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:36.065431 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:36.065466 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:36.067316 (+ 1850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.067684 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.067690 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.067694 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.067779 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.067940 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.067946 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.067950 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.068303 (+ 353us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:36.068468 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:36.068703 (+ 235us) server_negotiation.cc:300] Negotiation successful
0903 14:27:36.068917 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20250903 14:27:36.069667 28604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:36.070827 28610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4651ea0480f54ca891efbf00adc090a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } } }
I20250903 14:27:36.070941 28610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:36.071245 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4651ea0480f54ca891efbf00adc090a5. Latest consensus state: current_term: 1 leader_uuid: "4651ea0480f54ca891efbf00adc090a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } } }
I20250903 14:27:36.071354 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: This master's current role is: LEADER
W20250903 14:27:36.073518 28624 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:27:36.073585 28624 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:27:36.073650 28618 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:36.074219 28618 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:36.078269 28618 catalog_manager.cc:1349] Generated new cluster ID: af71f4beabbd4e17849cd208a4864a65
I20250903 14:27:36.078315 28618 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:36.086282 28618 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:36.086928 28618 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:36.092881 28618 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Generated new TSK 0
I20250903 14:27:36.093369 28618 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:36.137231 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44265
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:36.227139 28632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:36.227316 28632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:36.227397 28632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:36.230206 28632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:36.230255 28632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:36.230355 28632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:36.233979 28632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.234805 28632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.235527 28632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 14:27:36.240962 28632 server_base.cc:1047] running on GCE node
W20250903 14:27:36.240931 28640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.240923 28637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.240911 28638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.241518 28632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.241956 28632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.243088 28632 hybrid_clock.cc:648] HybridClock initialized: now 1756909656243076 us; error 25 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.245421 28632 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:27:36.246304 28632 webserver.cc:480] Webserver started at http://127.19.65.129:36197/ using document root <none> and password file <none>
I20250903 14:27:36.246786 28632 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.246837 28632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.246996 28632 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:36.248466 28632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "f231ea85ac0f4dad93c391d37e0ebe20"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "a72021f0f787512f67bc6478dbc18bfe"
server_key_iv: "d93a09b51ec43a36198cff6fa967abb5"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.248920 28632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "f231ea85ac0f4dad93c391d37e0ebe20"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "a72021f0f787512f67bc6478dbc18bfe"
server_key_iv: "d93a09b51ec43a36198cff6fa967abb5"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.251801 28632 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 14:27:36.253764 28647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.254611 28632 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:27:36.254698 28632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "f231ea85ac0f4dad93c391d37e0ebe20"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "a72021f0f787512f67bc6478dbc18bfe"
server_key_iv: "d93a09b51ec43a36198cff6fa967abb5"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.254773 28632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:36.274955 28632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:36.277150 28632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:36.277288 28632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:36.277757 28632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:36.278427 28632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:36.278468 28632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.278519 28632 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:36.278568 28632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.285858 28632 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42745
I20250903 14:27:36.285880 28760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:42745 every 8 connection(s)
I20250903 14:27:36.286633 28632 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:36.291219 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28632
I20250903 14:27:36.291312 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:36.291539 19718 external_mini_cluster.cc:1468] Setting key 8d0a0bdaddad7b054d964e52f1eba1d4
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:36.297535 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.288030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:58183 (local address 127.19.65.190:44265)
0903 14:27:36.288145 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:36.288148 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:36.288656 (+ 508us) server_negotiation.cc:408] Connection header received
0903 14:27:36.289320 (+ 664us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:36.289323 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:36.289389 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:36.289469 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:36.290515 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.290964 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.292052 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.292230 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.294082 (+ 1852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:36.294095 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:36.294097 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:36.294118 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:36.295670 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.296164 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.296170 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.296174 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.296228 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.296490 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.296495 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.296499 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.296698 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:36.296800 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:36.297166 (+ 366us) server_negotiation.cc:300] Negotiation successful
0903 14:27:36.297286 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:36.297971 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.288197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:58183)
0903 14:27:36.288549 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:36.288577 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:36.289158 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:36.289633 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:36.289639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:36.289935 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:36.290380 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.290391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.291079 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.291082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:36.291948 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.291959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.292128 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.292679 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:36.292699 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:36.293957 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:36.295779 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.295784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.295791 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.296064 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.296314 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.296316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.296317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.296390 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.296789 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:36.296793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:36.296949 (+ 156us) client_negotiation.cc:769] Sending connection context
0903 14:27:36.297079 (+ 130us) client_negotiation.cc:240] Negotiation successful
0903 14:27:36.297335 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20250903 14:27:36.298897 28761 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44265
I20250903 14:27:36.299085 28761 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:36.299485 28761 heartbeater.cc:507] Master 127.19.65.190:44265 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:27:36.300717 28568 ts_manager.cc:194] Registered new tserver with Master: f231ea85ac0f4dad93c391d37e0ebe20 (127.19.65.129:42745)
I20250903 14:27:36.301868 28568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:58183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:36.331460 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44265
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:36.419927 28768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:36.420070 28768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:36.420137 28768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:36.422932 28768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:36.422976 28768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:36.423038 28768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:36.426713 28768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.427433 28768 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.428054 28768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:36.433311 28773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.433301 28768 server_base.cc:1047] running on GCE node
W20250903 14:27:36.433387 28774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.433275 28776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.433832 28768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.434273 28768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.435417 28768 hybrid_clock.cc:648] HybridClock initialized: now 1756909656435413 us; error 27 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.437721 28768 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:27:36.438469 28768 webserver.cc:480] Webserver started at http://127.19.65.130:42991/ using document root <none> and password file <none>
I20250903 14:27:36.438863 28768 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.438902 28768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.439039 28768 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:36.440541 28768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "b2882b22d2044a2e97e59125ca550273"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "152c5c8840c29afef09c728af264aa38"
server_key_iv: "928d64b136a3744526dde7bc862411cb"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.440953 28768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "b2882b22d2044a2e97e59125ca550273"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "152c5c8840c29afef09c728af264aa38"
server_key_iv: "928d64b136a3744526dde7bc862411cb"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.443724 28768 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250903 14:27:36.445379 28783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.446277 28768 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:36.446362 28768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "b2882b22d2044a2e97e59125ca550273"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "152c5c8840c29afef09c728af264aa38"
server_key_iv: "928d64b136a3744526dde7bc862411cb"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.446453 28768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:36.461416 28768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:36.463551 28768 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:36.463685 28768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:36.464108 28768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:36.464752 28768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:36.464793 28768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.464831 28768 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:36.464860 28768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.472406 28768 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:44927
I20250903 14:27:36.472426 28896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:44927 every 8 connection(s)
I20250903 14:27:36.473093 28768 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:36.475507 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28768
I20250903 14:27:36.475632 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:36.475888 19718 external_mini_cluster.cc:1468] Setting key 3f0676a26ae8b0d4dab658a0d84e8012
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20250903 14:27:36.484953 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.474657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41797 (local address 127.19.65.190:44265)
0903 14:27:36.474785 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:36.474790 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:36.475460 (+ 670us) server_negotiation.cc:408] Connection header received
0903 14:27:36.476502 (+ 1042us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:36.476506 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:36.476572 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:36.476658 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:36.478003 (+ 1345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.478463 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.479477 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.479651 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.481910 (+ 2259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:36.481932 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:36.481936 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:36.481965 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:36.483332 (+ 1367us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.483834 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.483836 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.483838 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.483869 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.484115 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.484117 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.484118 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.484232 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:36.484292 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:36.484764 (+ 472us) server_negotiation.cc:300] Negotiation successful
0903 14:27:36.484851 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 14:27:36.485462 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.474865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:41797)
0903 14:27:36.475315 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:36.475361 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:36.476274 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:36.476895 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:36.476904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:36.477339 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:36.477878 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.477889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.478568 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.478571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:36.479369 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.479376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.479542 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.480112 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:36.480131 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:36.481751 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:36.483438 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.483443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.483452 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.483733 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.483939 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.483943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.483946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.484042 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.484316 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:36.484323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:36.484564 (+ 241us) client_negotiation.cc:769] Sending connection context
0903 14:27:36.484731 (+ 167us) client_negotiation.cc:240] Negotiation successful
0903 14:27:36.484897 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
I20250903 14:27:36.486510 28897 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44265
I20250903 14:27:36.486737 28897 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:36.487192 28897 heartbeater.cc:507] Master 127.19.65.190:44265 requested a full tablet report, sending...
I20250903 14:27:36.488019 28568 ts_manager.cc:194] Registered new tserver with Master: b2882b22d2044a2e97e59125ca550273 (127.19.65.130:44927)
I20250903 14:27:36.488535 28568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:41797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:36.514339 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44265
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:36.601850 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:36.601996 28904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:36.602068 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:36.604725 28904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:36.604770 28904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:36.604836 28904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:36.608527 28904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.609241 28904 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.609877 28904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:36.614807 28909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.614809 28912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.614866 28910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.615005 28904 server_base.cc:1047] running on GCE node
I20250903 14:27:36.615327 28904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.615783 28904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.616926 28904 hybrid_clock.cc:648] HybridClock initialized: now 1756909656616914 us; error 31 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.619122 28904 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:27:36.619866 28904 webserver.cc:480] Webserver started at http://127.19.65.131:44555/ using document root <none> and password file <none>
I20250903 14:27:36.620268 28904 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.620311 28904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.620500 28904 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:36.621987 28904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "1095467138d04293a738573f27213fb8"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "8088f164c7e945bb67278c0e3cea2576"
server_key_iv: "82b155dce165ae66081793d267644cdd"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.622360 28904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "1095467138d04293a738573f27213fb8"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "8088f164c7e945bb67278c0e3cea2576"
server_key_iv: "82b155dce165ae66081793d267644cdd"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.624960 28904 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250903 14:27:36.626825 28919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.627595 28904 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:36.627677 28904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "1095467138d04293a738573f27213fb8"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "8088f164c7e945bb67278c0e3cea2576"
server_key_iv: "82b155dce165ae66081793d267644cdd"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.627761 28904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:36.635437 28904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:36.637555 28904 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:36.637682 28904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:36.638089 28904 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:36.638741 28904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:36.638782 28904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.638831 28904 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:36.638859 28904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.646101 28904 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38587
I20250903 14:27:36.646126 29032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:38587 every 8 connection(s)
I20250903 14:27:36.646849 28904 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:36.647948 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 28904
I20250903 14:27:36.648041 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:36.648262 19718 external_mini_cluster.cc:1468] Setting key aaa2db4eedc36f914d0da62416c00f5c
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:36.656268 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.648102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:56895 (local address 127.19.65.190:44265)
0903 14:27:36.648233 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:36.648238 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:36.648759 (+ 521us) server_negotiation.cc:408] Connection header received
0903 14:27:36.649590 (+ 831us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:36.649593 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:36.649641 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:36.649713 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:36.650677 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.651100 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.651952 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.652068 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.653738 (+ 1670us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:36.653752 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:36.653754 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:36.653776 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:36.654976 (+ 1200us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.655296 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.655298 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.655299 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.655350 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:36.655550 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:36.655552 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:36.655553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:36.655673 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:36.655739 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:36.656070 (+ 331us) server_negotiation.cc:300] Negotiation successful
0903 14:27:36.656160 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:36.656798 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:36.648285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:56895)
0903 14:27:36.648653 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:36.648688 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:36.649366 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:36.649806 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:36.649815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:36.650121 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:36.650568 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.650578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.651180 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:36.651182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:36.651872 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:36.651878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:36.651995 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:36.652410 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:36.652427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:36.653622 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:36.655055 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.655058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.655064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.655232 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.655427 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:36.655430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:36.655431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:36.655493 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:36.655738 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:36.655741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:36.655902 (+ 161us) client_negotiation.cc:769] Sending connection context
0903 14:27:36.656044 (+ 142us) client_negotiation.cc:240] Negotiation successful
0903 14:27:36.656209 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20250903 14:27:36.657634 29033 heartbeater.cc:344] Connected to a master server at 127.19.65.190:44265
I20250903 14:27:36.657804 29033 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:36.658232 29033 heartbeater.cc:507] Master 127.19.65.190:44265 requested a full tablet report, sending...
I20250903 14:27:36.659044 28568 ts_manager.cc:194] Registered new tserver with Master: 1095467138d04293a738573f27213fb8 (127.19.65.131:38587)
I20250903 14:27:36.659510 28568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:56895
I20250903 14:27:36.661273 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
I20250903 14:27:36.715509 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28632
I20250903 14:27:36.720623 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28768
I20250903 14:27:36.725229 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28904
I20250903 14:27:36.729831 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 28536
I20250903 14:27:36.735172 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44265
--webserver_interface=127.19.65.190
--webserver_port=38355
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:36.826725 29042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:36.826896 29042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:36.826941 29042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:36.829659 29042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:36.829701 29042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:36.829721 29042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:36.829751 29042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:36.829780 29042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:36.833062 29042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=38355
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.833875 29042 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.834661 29042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:36.840498 29050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.840507 29048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.840651 29042 server_base.cc:1047] running on GCE node
W20250903 14:27:36.840765 29047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.841205 29042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.841928 29042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.843063 29042 hybrid_clock.cc:648] HybridClock initialized: now 1756909656843042 us; error 32 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.845340 29042 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:27:36.846271 29042 webserver.cc:480] Webserver started at http://127.19.65.190:38355/ using document root <none> and password file <none>
I20250903 14:27:36.846758 29042 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.846814 29042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.849525 29042 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:27:36.851001 29057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.851783 29042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:36.851872 29042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "4651ea0480f54ca891efbf00adc090a5"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "c4f1618614f56e78df2f79bf6dd109a7"
server_key_iv: "bd768e956f2003b99e10e674e128675a"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.852160 29042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:36.859872 29042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:36.861884 29042 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:36.862015 29042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:36.867861 29109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:44265 every 8 connection(s)
I20250903 14:27:36.867862 29042 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44265
I20250903 14:27:36.868775 29042 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:36.869426 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29042
I20250903 14:27:36.869884 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:42745
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:44265
--webserver_port=36197
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.129
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 14:27:36.872697 29110 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:27:36.874032 29110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Bootstrap starting.
I20250903 14:27:36.887959 29110 log.cc:826] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:36.893087 29110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:27:36.893452 29110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Bootstrap complete.
I20250903 14:27:36.896420 29110 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.896715 29110 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4651ea0480f54ca891efbf00adc090a5, State: Initialized, Role: FOLLOWER
I20250903 14:27:36.897082 29110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.897188 29110 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:36.897241 29110 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:36.897326 29110 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:27:36.898538 29110 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.898789 29110 leader_election.cc:304] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4651ea0480f54ca891efbf00adc090a5; no voters:
I20250903 14:27:36.899003 29110 leader_election.cc:290] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250903 14:27:36.899070 29114 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 14:27:36.899214 29114 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [term 2 LEADER]: Becoming Leader. State: Replica: 4651ea0480f54ca891efbf00adc090a5, State: Running, Role: LEADER
I20250903 14:27:36.899447 29114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } }
I20250903 14:27:36.899885 29110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:36.900139 29115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4651ea0480f54ca891efbf00adc090a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } } }
I20250903 14:27:36.900250 29115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:36.900156 29114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4651ea0480f54ca891efbf00adc090a5. Latest consensus state: current_term: 2 leader_uuid: "4651ea0480f54ca891efbf00adc090a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4651ea0480f54ca891efbf00adc090a5" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44265 } } }
I20250903 14:27:36.900352 29114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:36.900585 29118 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:36.903759 29118 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:36.904282 29118 catalog_manager.cc:1261] Loaded cluster ID: af71f4beabbd4e17849cd208a4864a65
I20250903 14:27:36.904340 29118 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:36.906549 29118 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:36.907147 29118 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4651ea0480f54ca891efbf00adc090a5: Loaded TSK: 0
I20250903 14:27:36.908181 29118 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250903 14:27:36.960955 29112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:36.961112 29112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:36.961194 29112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:36.963923 29112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:36.963972 29112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:36.964056 29112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:36.967644 29112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:42745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=36197
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:36.968367 29112 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:36.968994 29112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:36.974283 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.974395 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:36.974287 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:36.974690 29112 server_base.cc:1047] running on GCE node
I20250903 14:27:36.974967 29112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:36.975385 29112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:36.976542 29112 hybrid_clock.cc:648] HybridClock initialized: now 1756909656976520 us; error 40 us; skew 500 ppm
Sep 03 14:27:36 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909656, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:36.978922 29112 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20250903 14:27:36.979946 29112 webserver.cc:480] Webserver started at http://127.19.65.129:36197/ using document root <none> and password file <none>
I20250903 14:27:36.980494 29112 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:36.980561 29112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:36.983987 29112 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:36.985831 29146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:36.986846 29112 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:36.986935 29112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "f231ea85ac0f4dad93c391d37e0ebe20"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "a72021f0f787512f67bc6478dbc18bfe"
server_key_iv: "d93a09b51ec43a36198cff6fa967abb5"
server_key_version: "encryptionkey@0"
I20250903 14:27:36.987287 29112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:37.008008 29112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:37.010696 29112 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:37.010846 29112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:37.011332 29112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:37.012145 29112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:37.012197 29112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.012246 29112 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:37.012271 29112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.021998 29112 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42745
I20250903 14:27:37.022007 29259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:42745 every 8 connection(s)
I20250903 14:27:37.023057 29112 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:37.024483 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29112
I20250903 14:27:37.024968 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:44927
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:44265
--webserver_port=42991
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.130
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.036589 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.024931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:35297)
0903 14:27:37.025366 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.025446 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.026294 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.028114 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.028124 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.028612 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.029266 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.029281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.031110 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.031116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.031728 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.031740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.031924 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.033198 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.033235 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.035900 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
W20250903 14:27:37.037128 29260 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:44265 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:44265: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
W20250903 14:27:37.037386 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.026622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:35297 (local address 127.19.65.190:44265)
0903 14:27:37.027061 (+ 439us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.027071 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.027105 (+ 34us) server_negotiation.cc:408] Connection header received
0903 14:27:37.027343 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.027356 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.027694 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.029514 (+ 1820us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.029810 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.030946 (+ 1136us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.031853 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.032084 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.036120 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:35297: BlockingRecv error: recv got EOF from 127.19.65.129:35297 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.042330 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.037504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:57125)
0903 14:27:37.037913 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.037926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.037998 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.038290 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.038293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.038437 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.038638 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.038645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.039483 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.039486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.039924 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.039932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.040044 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.040503 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.040516 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.042229 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345}
W20250903 14:27:37.042697 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.037703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:57125 (local address 127.19.65.190:44265)
0903 14:27:37.037895 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.037899 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.037981 (+ 82us) server_negotiation.cc:408] Connection header received
0903 14:27:37.038129 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.038133 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.038172 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.038254 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.038752 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.039370 (+ 618us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.040051 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.040577 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.042359 (+ 1782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:57125: BlockingRecv error: recv got EOF from 127.19.65.129:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.047116 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.042763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:37429)
0903 14:27:37.043013 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.043024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.043090 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.043291 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.043293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.043413 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.043568 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.043573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.044353 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.044357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.044772 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.044780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.044874 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.045277 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.045290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.047034 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197}
W20250903 14:27:37.047217 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.042922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37429 (local address 127.19.65.190:44265)
0903 14:27:37.043010 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.043014 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.043070 (+ 56us) server_negotiation.cc:408] Connection header received
0903 14:27:37.043170 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.043173 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.043206 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.043268 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.043658 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.044254 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.044868 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.045295 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.047142 (+ 1847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:37429: BlockingRecv error: recv got EOF from 127.19.65.129:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250903 14:27:37.047595 29260 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250903 14:27:37.117950 29263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:37.118103 29263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:37.118209 29263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:37.120936 29263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:37.120986 29263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:37.121083 29263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:37.124387 29263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=42991
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:37.125144 29263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:37.125866 29263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:37.130929 29273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:37.130941 29270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:37.130944 29271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:37.131151 29263 server_base.cc:1047] running on GCE node
I20250903 14:27:37.131503 29263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:37.131928 29263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:37.133055 29263 hybrid_clock.cc:648] HybridClock initialized: now 1756909657133045 us; error 25 us; skew 500 ppm
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909657, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:37.135185 29263 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20250903 14:27:37.136018 29263 webserver.cc:480] Webserver started at http://127.19.65.130:42991/ using document root <none> and password file <none>
I20250903 14:27:37.136487 29263 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:37.136554 29263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:37.139389 29263 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:37.140865 29280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.141737 29263 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:37.141824 29263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "b2882b22d2044a2e97e59125ca550273"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "152c5c8840c29afef09c728af264aa38"
server_key_iv: "928d64b136a3744526dde7bc862411cb"
server_key_version: "encryptionkey@0"
I20250903 14:27:37.142107 29263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:37.164031 29263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:37.166491 29263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:37.166664 29263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:37.167063 29263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:37.167702 29263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:37.167744 29263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.167793 29263 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:37.167819 29263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.175057 29263 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:44927
I20250903 14:27:37.175103 29393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:44927 every 8 connection(s)
I20250903 14:27:37.175952 29263 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:37.179941 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29263
I20250903 14:27:37.180316 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:38587
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:44265
--webserver_port=44555
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.131
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20250903 14:27:37.184083 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.177312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:37837 (local address 127.19.65.190:44265)
0903 14:27:37.177416 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.177419 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.177950 (+ 531us) server_negotiation.cc:408] Connection header received
0903 14:27:37.178616 (+ 666us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.178620 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.178676 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.178744 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.179712 (+ 968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.180183 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.180775 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.180934 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.184001 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:37837: BlockingRecv error: recv got EOF from 127.19.65.130:37837 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:27:37.184564 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.177497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:37837)
0903 14:27:37.177835 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.177863 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.178449 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.178877 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.178883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.179168 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.179587 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.179597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.180282 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.180284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.180675 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.180682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.180834 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.181786 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.181804 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.183902 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
W20250903 14:27:37.184875 29394 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:44265 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:44265: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.188884 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.185205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:49805)
0903 14:27:37.185287 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.185299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.185364 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.185660 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.185663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.185794 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.185950 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.185955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.186624 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.186627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.186960 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.186965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.187052 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.187507 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.187518 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.188790 (+ 1272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20250903 14:27:37.188994 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.185291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:49805 (local address 127.19.65.190:44265)
0903 14:27:37.185427 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.185448 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.185463 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:37.185519 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.185522 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.185566 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.185633 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.186060 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.186510 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.187051 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.187226 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.188901 (+ 1675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:49805: BlockingRecv error: recv got EOF from 127.19.65.130:49805 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.192773 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.189355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:34197)
0903 14:27:37.189429 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.189449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.189556 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.189739 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.189742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.189864 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.189977 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.189981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.190561 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.190565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.190848 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.190853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.190939 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.191398 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.191412 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.192671 (+ 1259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20250903 14:27:37.192842 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.189439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34197 (local address 127.19.65.190:44265)
0903 14:27:37.189573 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.189578 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.189592 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:37.189624 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.189627 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.189658 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.189721 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.190071 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.190471 (+ 400us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.190937 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.191109 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.192763 (+ 1654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:34197: BlockingRecv error: recv got EOF from 127.19.65.130:34197 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20250903 14:27:37.192912 29394 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250903 14:27:37.268436 29397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:37.268581 29397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:37.268625 29397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:37.271257 29397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:37.271301 29397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:37.271363 29397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:37.274740 29397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:38587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=44555
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:37.275471 29397 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:37.276149 29397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:37.281081 29403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:37.281162 29405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:37.281150 29402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:37.281270 29397 server_base.cc:1047] running on GCE node
I20250903 14:27:37.281595 29397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:37.282014 29397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:37.283140 29397 hybrid_clock.cc:648] HybridClock initialized: now 1756909657283118 us; error 36 us; skew 500 ppm
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909657, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:37.285233 29397 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20250903 14:27:37.286095 29397 webserver.cc:480] Webserver started at http://127.19.65.131:44555/ using document root <none> and password file <none>
I20250903 14:27:37.286528 29397 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:37.286582 29397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:37.289131 29397 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:37.290617 29412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.291357 29397 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:37.291435 29397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "1095467138d04293a738573f27213fb8"
format_stamp: "Formatted at 2025-09-03 14:27:36 on dist-test-slave-4wpb"
server_key: "8088f164c7e945bb67278c0e3cea2576"
server_key_iv: "82b155dce165ae66081793d267644cdd"
server_key_version: "encryptionkey@0"
I20250903 14:27:37.291738 29397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:37.306955 29397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:37.309012 29397 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:37.309149 29397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:37.309639 29397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:37.310281 29397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:37.310322 29397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.310360 29397 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:37.310390 29397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:37.317961 29397 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38587
I20250903 14:27:37.317988 29525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:38587 every 8 connection(s)
I20250903 14:27:37.318763 29397 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:37.324913 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29397
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20250903 14:27:37.327495 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.320184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57767 (local address 127.19.65.190:44265)
0903 14:27:37.320303 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.320306 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.320867 (+ 561us) server_negotiation.cc:408] Connection header received
0903 14:27:37.321611 (+ 744us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.321615 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.321675 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.321752 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.322841 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.323300 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.323850 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.324021 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.327412 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:57767: BlockingRecv error: recv got EOF from 127.19.65.131:57767 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 14:27:37.327945 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.320408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:57767)
0903 14:27:37.320739 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.320766 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.321397 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.321901 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.321908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.322237 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.322693 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.322703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.323402 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.323405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.323751 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.323757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.323913 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.324851 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.324878 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.327315 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
W20250903 14:27:37.328428 29526 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:44265 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:44265: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.333230 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.329001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:46471)
0903 14:27:37.329139 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.329179 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.329233 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.329453 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.329454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.329589 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.329798 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.329827 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.330555 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.330561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.331023 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.331036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.331146 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.331535 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.331554 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.333098 (+ 1544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20250903 14:27:37.333334 29264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.329098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46471 (local address 127.19.65.190:44265)
0903 14:27:37.329254 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.329259 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.329273 (+ 14us) server_negotiation.cc:408] Connection header received
0903 14:27:37.329319 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.329321 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.329364 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.329440 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.329937 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.330452 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.331134 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.331252 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.333238 (+ 1986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:46471: BlockingRecv error: recv got EOF from 127.19.65.131:46471 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20250903 14:27:37.333361 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.325774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.19.65.190:44265)
0903 14:27:37.325925 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.325929 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.325945 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:27:37.326012 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.326015 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.326062 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.326170 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.327096 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.327777 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.328462 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.328636 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.329732 (+ 1096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:37.329749 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:37.329760 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:37.329801 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:37.331705 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:37.332088 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:37.332091 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:37.332095 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:37.332158 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:37.332385 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:37.332388 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:37.332389 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:37.332672 (+ 283us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:37.332827 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:37.333052 (+ 225us) server_negotiation.cc:300] Negotiation successful
0903 14:27:37.333257 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:37 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:37.337302 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:37.333632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:40871)
0903 14:27:37.333708 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:37.333720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:37.333776 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:37.334341 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:37.334344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:37.334454 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:37.334565 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.334569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.335118 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.335121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:37.335399 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:37.335404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.335547 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.335924 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:37.335935 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:37.337203 (+ 1268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20250903 14:27:37.337378 29265 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:37.334005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40871 (local address 127.19.65.190:44265)
0903 14:27:37.334137 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:37.334142 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:37.334157 (+ 15us) server_negotiation.cc:408] Connection header received
0903 14:27:37.334191 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:37.334196 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:37.334236 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:37.334305 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:37.334663 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.335027 (+ 364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:37.335541 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:37.335733 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:37.337292 (+ 1559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:40871: BlockingRecv error: recv got EOF from 127.19.65.131:40871 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20250903 14:27:37.337525 29526 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:38.052452 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:38.048162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:57933)
0903 14:27:38.048395 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:38.048406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:38.048480 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:38.048686 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:38.048688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:38.048852 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:38.049028 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.049032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.049823 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.049827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:38.050290 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.050298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.050377 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.050831 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:38.050842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:38.052321 (+ 1479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
W20250903 14:27:38.052541 29531 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:38.048255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:57933 (local address 127.19.65.190:44265)
0903 14:27:38.048444 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:38.048447 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:38.048459 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:38.048565 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:38.048567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:38.048605 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:38.048670 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:38.049112 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.049719 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.050378 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.050499 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.052428 (+ 1929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:57933: BlockingRecv error: recv got EOF from 127.19.65.129:57933 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":49,"threads_started":1}
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:38.197681 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:38.193395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:50511)
0903 14:27:38.193630 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:38.193642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:38.193734 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:38.193928 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:38.193931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:38.194085 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:38.194312 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.194323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.195004 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.195007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:38.195346 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.195352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.195446 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.196014 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:38.196027 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:38.197591 (+ 1564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20250903 14:27:38.197783 29531 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:38.193514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50511 (local address 127.19.65.190:44265)
0903 14:27:38.193642 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:38.193645 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:38.193700 (+ 55us) server_negotiation.cc:408] Connection header received
0903 14:27:38.193800 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:38.193803 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:38.193844 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:38.193911 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:38.194453 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.194914 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.195443 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.195678 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.197697 (+ 2019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:50511: BlockingRecv error: recv got EOF from 127.19.65.130:50511 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:38 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:38.342319 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:38.338016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:41919)
0903 14:27:38.338247 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:38.338261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:38.338355 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:38.338619 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:38.338639 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:38.338801 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:38.338971 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.338976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.339761 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.339764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:38.340098 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:38.340104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.340196 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.340670 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:38.340681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:38.342191 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
W20250903 14:27:38.342403 29531 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:38.338113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:41919 (local address 127.19.65.190:44265)
0903 14:27:38.338238 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:38.338243 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:38.338320 (+ 77us) server_negotiation.cc:408] Connection header received
0903 14:27:38.338458 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:38.338460 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:38.338522 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:38.338591 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:38.339073 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.339660 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:38.340187 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:38.340365 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:38.342312 (+ 1947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:41919: BlockingRecv error: recv got EOF from 127.19.65.131:41919 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:39.057621 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:39.053131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:53559)
0903 14:27:39.053376 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:39.053387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:39.053461 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:39.053712 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:39.053714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:39.053859 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:39.054028 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.054032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.054796 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.054799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:39.055258 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.055265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.055377 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.055885 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:39.055899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:39.057522 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":61,"threads_started":1}
W20250903 14:27:39.057725 29535 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:39.053320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53559 (local address 127.19.65.190:44265)
0903 14:27:39.053539 (+ 219us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:39.053542 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:39.053555 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:39.053583 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:39.053585 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:39.053621 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:39.053685 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:39.054103 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.054702 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.055382 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.055561 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.057627 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:53559: BlockingRecv error: recv got EOF from 127.19.65.129:53559 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:39.227344 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:39.198275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:58535)
0903 14:27:39.198507 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:39.198519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:39.198618 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:39.198833 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:39.198835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:39.199001 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:39.199220 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.199226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.199959 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.199962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:39.200295 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.200300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.200397 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.225580 (+ 25183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:39.225592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:39.227219 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
W20250903 14:27:39.227451 29535 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:39.198351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:58535 (local address 127.19.65.190:44265)
0903 14:27:39.198480 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:39.198485 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:39.198589 (+ 104us) server_negotiation.cc:408] Connection header received
0903 14:27:39.198702 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:39.198705 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:39.198745 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:39.198803 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:39.199352 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.199868 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.200391 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.200586 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.227362 (+ 26776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:58535: BlockingRecv error: recv got EOF from 127.19.65.130:58535 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:39 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:39.347425 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:39.342908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:60245)
0903 14:27:39.343154 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:39.343165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:39.343253 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:39.343475 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:39.343478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:39.343638 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:39.343806 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.343812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.344588 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.344591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:39.344967 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:39.344974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.345081 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.345605 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:39.345618 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:39.347308 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20250903 14:27:39.347555 29535 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:39.343028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:60245 (local address 127.19.65.190:44265)
0903 14:27:39.343154 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:39.343156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:39.343220 (+ 64us) server_negotiation.cc:408] Connection header received
0903 14:27:39.343339 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:39.343342 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:39.343386 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:39.343445 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:39.343906 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.344450 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:39.345073 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:39.345255 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:39.347452 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:60245: BlockingRecv error: recv got EOF from 127.19.65.131:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:40.062507 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:40.058191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:37907)
0903 14:27:40.058477 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:40.058491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:40.058593 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:40.058849 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:40.058852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:40.059033 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:40.059211 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.059217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.059906 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.059909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:40.060235 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.060241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.060334 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.060903 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:40.060915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:40.062374 (+ 1459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":73,"threads_started":1}
W20250903 14:27:40.062623 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:40.058262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37907 (local address 127.19.65.190:44265)
0903 14:27:40.058526 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:40.058529 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:40.058541 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:40.058701 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:40.058705 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:40.058748 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:40.058817 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:40.059325 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.059802 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.060331 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.060527 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.062520 (+ 1993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:37907: BlockingRecv error: recv got EOF from 127.19.65.129:37907 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:40.231753 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:40.227987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:48581)
0903 14:27:40.228209 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:40.228220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:40.228304 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:40.228504 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:40.228506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:40.228648 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:40.228818 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.228822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.229374 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.229376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:40.229717 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.229722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.229791 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.230239 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:40.230249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:40.231660 (+ 1411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":63,"threads_started":1}
W20250903 14:27:40.231889 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:40.228049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:48581 (local address 127.19.65.190:44265)
0903 14:27:40.228201 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:40.228204 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:40.228277 (+ 73us) server_negotiation.cc:408] Connection header received
0903 14:27:40.228367 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:40.228369 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:40.228420 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:40.228477 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:40.228891 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.229303 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.229795 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.229911 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.231773 (+ 1862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:48581: BlockingRecv error: recv got EOF from 127.19.65.130:48581 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:40 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:40.352172 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:40.347988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:40847)
0903 14:27:40.348247 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:40.348261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:40.348380 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:40.348608 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:40.348611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:40.348787 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:40.349006 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.349012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.349726 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.349729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:40.350077 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:40.350083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.350181 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.350663 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:40.350673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:40.352036 (+ 1363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
W20250903 14:27:40.352238 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:40.348059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40847 (local address 127.19.65.190:44265)
0903 14:27:40.348211 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:40.348217 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:40.348337 (+ 120us) server_negotiation.cc:408] Connection header received
0903 14:27:40.348479 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:40.348482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:40.348525 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:40.348583 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:40.349108 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.349626 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:40.350178 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:40.350361 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:40.352144 (+ 1783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:40847: BlockingRecv error: recv got EOF from 127.19.65.131:40847 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:41.067270 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:41.063148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:41199)
0903 14:27:41.063397 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:41.063409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:41.063508 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:41.063706 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:41.063708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:41.063859 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:41.064034 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.064039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.064636 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.064638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:41.064979 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.064984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.065053 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.065563 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:41.065575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:41.067160 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
W20250903 14:27:41.067394 29543 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:41.063198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:41199 (local address 127.19.65.190:44265)
0903 14:27:41.063418 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:41.063421 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:41.063477 (+ 56us) server_negotiation.cc:408] Connection header received
0903 14:27:41.063576 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:41.063578 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:41.063618 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:41.063693 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:41.064115 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.064547 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.065052 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.065162 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.067287 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:41199: BlockingRecv error: recv got EOF from 127.19.65.129:41199 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:41.236441 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:41.232376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:44969)
0903 14:27:41.232625 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:41.232637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:41.232731 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:41.232913 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:41.232915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:41.233068 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:41.233240 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.233245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.233919 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.233921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:41.234255 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.234260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.234340 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.234848 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:41.234860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:41.236333 (+ 1473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20250903 14:27:41.236560 29543 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:41.232452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:44969 (local address 127.19.65.190:44265)
0903 14:27:41.232562 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:41.232565 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:41.232691 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:27:41.232793 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:41.232795 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:41.232834 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:41.232891 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:41.233314 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.233833 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.234338 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.234461 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.236449 (+ 1988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:44969: BlockingRecv error: recv got EOF from 127.19.65.130:44969 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:41 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:41.357113 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:41.352811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:54211)
0903 14:27:41.353096 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:41.353110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:41.353222 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:41.353499 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:41.353502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:41.353696 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:41.353879 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.353884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.354495 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.354497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:41.354823 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:41.354827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.354904 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.355378 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:41.355389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:41.357008 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
W20250903 14:27:41.357254 29543 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:41.352891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:54211 (local address 127.19.65.190:44265)
0903 14:27:41.353024 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:41.353029 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:41.353194 (+ 165us) server_negotiation.cc:408] Connection header received
0903 14:27:41.353309 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:41.353313 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:41.353360 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:41.353459 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:41.353972 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.354411 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:41.354896 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:41.355007 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:41.357133 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:54211: BlockingRecv error: recv got EOF from 127.19.65.131:54211 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:42.071906 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:42.067925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:52543)
0903 14:27:42.068152 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:42.068165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:42.068249 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:42.068460 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:42.068462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:42.068615 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:42.068804 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.068809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.069413 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.069414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:42.069764 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.069770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.069845 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.070319 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:42.070330 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:42.071804 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
W20250903 14:27:42.072050 29547 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:42.067974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:52543 (local address 127.19.65.190:44265)
0903 14:27:42.068182 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:42.068185 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:42.068212 (+ 27us) server_negotiation.cc:408] Connection header received
0903 14:27:42.068322 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:42.068325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:42.068364 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:42.068432 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:42.068893 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.069338 (+ 445us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.069847 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.069981 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.071930 (+ 1949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:52543: BlockingRecv error: recv got EOF from 127.19.65.129:52543 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:42.240860 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:42.237096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:39663)
0903 14:27:42.237367 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:42.237378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:42.237455 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:42.237664 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:42.237666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:42.237817 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:42.237987 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.237992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.238557 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.238558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:42.238875 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.238880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.238950 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.239385 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:42.239395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:42.240767 (+ 1372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
W20250903 14:27:42.240959 29547 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:42.237217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:39663 (local address 127.19.65.190:44265)
0903 14:27:42.237363 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:42.237366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:42.237434 (+ 68us) server_negotiation.cc:408] Connection header received
0903 14:27:42.237537 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:42.237539 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:42.237577 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:42.237633 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:42.238069 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.238481 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.238949 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.239060 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.240872 (+ 1812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:39663: BlockingRecv error: recv got EOF from 127.19.65.130:39663 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:42 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:42.361856 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:42.357778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:44709)
0903 14:27:42.358025 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:42.358038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:42.358129 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:42.358348 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:42.358351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:42.358521 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:42.358688 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.358693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.359296 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.359299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:42.359623 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:42.359629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.359719 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.360266 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:42.360277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:42.361761 (+ 1484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
W20250903 14:27:42.361985 29547 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:42.357864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:44709 (local address 127.19.65.190:44265)
0903 14:27:42.357991 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:42.357995 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:42.358095 (+ 100us) server_negotiation.cc:408] Connection header received
0903 14:27:42.358209 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:42.358213 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:42.358252 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:42.358311 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:42.358797 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.359202 (+ 405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:42.359717 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:42.359904 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:42.361872 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:44709: BlockingRecv error: recv got EOF from 127.19.65.131:44709 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:43.076826 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:43.072512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:55239)
0903 14:27:43.072768 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:43.072779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:43.072866 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:43.073062 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:43.073064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:43.073217 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:43.073399 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.073403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.074122 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.074124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:43.074466 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.074471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.074557 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.075100 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:43.075113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:43.076711 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
W20250903 14:27:43.076968 29551 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:43.072558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:55239 (local address 127.19.65.190:44265)
0903 14:27:43.072783 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:43.072786 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:43.072824 (+ 38us) server_negotiation.cc:408] Connection header received
0903 14:27:43.072932 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:43.072934 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:43.072973 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:43.073041 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:43.073501 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.074041 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.074537 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.074659 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.076832 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:55239: BlockingRecv error: recv got EOF from 127.19.65.129:55239 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:43.245357 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:43.241501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:36201)
0903 14:27:43.241744 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:43.241756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:43.241841 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:43.242025 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:43.242026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:43.242173 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:43.242345 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.242350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.242934 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.242935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:43.243245 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.243249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.243320 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.243791 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:43.243803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:43.245261 (+ 1458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20250903 14:27:43.245502 29551 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:43.241566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:36201 (local address 127.19.65.190:44265)
0903 14:27:43.241684 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:43.241687 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:43.241813 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:27:43.241899 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:43.241901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:43.241939 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:43.242001 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:43.242431 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.242864 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.243310 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.243411 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.245367 (+ 1956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:36201: BlockingRecv error: recv got EOF from 127.19.65.130:36201 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:43 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:43.367084 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:43.362491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:42601)
0903 14:27:43.362778 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:43.362790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:43.362899 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:43.363150 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:43.363153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:43.363326 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:43.363500 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.363505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.364189 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.364193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:43.364582 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:43.364588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.364703 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.365257 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:43.365269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:43.366932 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
W20250903 14:27:43.367169 29551 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:43.362587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:42601 (local address 127.19.65.190:44265)
0903 14:27:43.362724 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:43.362728 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:43.362861 (+ 133us) server_negotiation.cc:408] Connection header received
0903 14:27:43.363003 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:43.363008 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:43.363051 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:43.363114 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:43.363608 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.364088 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:43.364698 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:43.364906 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:43.367060 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:42601: BlockingRecv error: recv got EOF from 127.19.65.131:42601 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:44.081729 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:44.077441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:41587)
0903 14:27:44.077765 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:44.077777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:44.077866 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:44.078070 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:44.078072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:44.078241 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:44.078440 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.078445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.079114 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.079117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:44.079469 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.079474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.079553 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.080053 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:44.080065 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:44.081628 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":94,"threads_started":1}
W20250903 14:27:44.081888 29555 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:44.077541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:41587 (local address 127.19.65.190:44265)
0903 14:27:44.077774 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:44.077778 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:44.077835 (+ 57us) server_negotiation.cc:408] Connection header received
0903 14:27:44.077939 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:44.077941 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:44.077981 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:44.078047 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:44.078550 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.079038 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.079551 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.079673 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.081744 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:41587: BlockingRecv error: recv got EOF from 127.19.65.129:41587 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:44.256011 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:44.245951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:50257)
0903 14:27:44.246197 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:44.246208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:44.246292 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:44.246516 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:44.246518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:44.246677 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:44.246862 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.246867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.247514 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.247516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:44.247848 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.247852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.247930 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.254260 (+ 6330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:44.254276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:44.255916 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
W20250903 14:27:44.256122 29555 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:44.246007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50257 (local address 127.19.65.190:44265)
0903 14:27:44.246128 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:44.246131 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:44.246264 (+ 133us) server_negotiation.cc:408] Connection header received
0903 14:27:44.246356 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:44.246358 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:44.246417 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:44.246481 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:44.246963 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.247439 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.247929 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.248040 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.256028 (+ 7988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:50257: BlockingRecv error: recv got EOF from 127.19.65.130:50257 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:44 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:44.371765 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:44.367728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:33387)
0903 14:27:44.367993 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:44.368004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:44.368088 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:44.368271 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:44.368273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:44.368409 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:44.368577 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.368582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.369140 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.369142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:44.369486 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:44.369490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.369581 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.370060 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:44.370071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:44.371672 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
W20250903 14:27:44.371840 29555 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:44.367800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:33387 (local address 127.19.65.190:44265)
0903 14:27:44.367904 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:44.367907 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:44.368056 (+ 149us) server_negotiation.cc:408] Connection header received
0903 14:27:44.368153 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:44.368155 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:44.368192 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:44.368249 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:44.368660 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.369071 (+ 411us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:44.369570 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:44.369690 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:44.371760 (+ 2070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:33387: BlockingRecv error: recv got EOF from 127.19.65.131:33387 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:45.086503 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:45.082384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:36375)
0903 14:27:45.082641 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:45.082652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:45.082731 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:45.082997 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:45.082999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:45.083144 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:45.083319 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.083323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.083985 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.083987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:45.084320 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.084324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.084398 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.084879 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:45.084889 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:45.086387 (+ 1498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":65,"threads_started":1}
W20250903 14:27:45.086679 29559 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:45.082590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36375 (local address 127.19.65.190:44265)
0903 14:27:45.082781 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:45.082785 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:45.082796 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:45.082826 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:45.082828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:45.082867 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:45.082940 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:45.083396 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.083900 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.084401 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.084525 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.086527 (+ 2002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:36375: BlockingRecv error: recv got EOF from 127.19.65.129:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:45.261157 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:45.256701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:51093)
0903 14:27:45.257001 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:45.257014 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:45.257134 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:45.257408 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:45.257415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:45.257651 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:45.257909 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.257923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.258697 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.258699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:45.259043 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.259048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.259126 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.259614 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:45.259626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:45.261057 (+ 1431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
W20250903 14:27:45.261289 29559 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:45.256781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51093 (local address 127.19.65.190:44265)
0903 14:27:45.256947 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:45.256954 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:45.257101 (+ 147us) server_negotiation.cc:408] Connection header received
0903 14:27:45.257232 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:45.257235 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:45.257280 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:45.257363 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:45.258010 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.258615 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.259126 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.259247 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.261162 (+ 1915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:51093: BlockingRecv error: recv got EOF from 127.19.65.130:51093 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:45 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:45.376653 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:45.372517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:50959)
0903 14:27:45.372764 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:45.372776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:45.372859 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:45.373042 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:45.373044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:45.373190 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:45.373359 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.373365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.374045 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.374048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:45.374401 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:45.374407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.374489 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.375010 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:45.375023 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:45.376548 (+ 1525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
W20250903 14:27:45.376799 29559 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:45.372576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:50959 (local address 127.19.65.190:44265)
0903 14:27:45.372689 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:45.372692 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:45.372829 (+ 137us) server_negotiation.cc:408] Connection header received
0903 14:27:45.372922 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:45.372924 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:45.372963 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:45.373024 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:45.373433 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.373951 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:45.374486 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:45.374606 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:45.376661 (+ 2055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:50959: BlockingRecv error: recv got EOF from 127.19.65.131:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:46.091547 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:46.087179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:44521)
0903 14:27:46.087457 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:46.087469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:46.087553 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:46.087812 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:46.087814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:46.087960 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:46.088146 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.088151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.088731 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.088733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:46.089085 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.089093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.089200 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.089758 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:46.089772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:46.091411 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
W20250903 14:27:46.091629 29563 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:46.087270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44521 (local address 127.19.65.190:44265)
0903 14:27:46.087503 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:46.087506 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:46.087519 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:46.087637 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:46.087639 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:46.087682 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:46.087770 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:46.088227 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.088659 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.089183 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.089342 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.091523 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44521: BlockingRecv error: recv got EOF from 127.19.65.129:44521 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:46.266077 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:46.261842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:57969)
0903 14:27:46.262095 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:46.262107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:46.262209 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:46.262427 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:46.262430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:46.262598 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:46.262779 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.262784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.263430 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.263433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:46.263779 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.263785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.263879 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.264506 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:46.264517 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:46.265983 (+ 1466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20250903 14:27:46.266204 29563 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:46.261926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57969 (local address 127.19.65.190:44265)
0903 14:27:46.262069 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:46.262073 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:46.262174 (+ 101us) server_negotiation.cc:408] Connection header received
0903 14:27:46.262297 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:46.262299 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:46.262339 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:46.262398 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:46.262884 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.263312 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.263877 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.264086 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.266103 (+ 2017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:57969: BlockingRecv error: recv got EOF from 127.19.65.130:57969 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:46 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:46.381613 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:46.377325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:49491)
0903 14:27:46.377612 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:46.377627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:46.377730 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:46.377977 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:46.377980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:46.378146 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:46.378331 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.378335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.378925 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.378928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:46.379288 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:46.379296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.379400 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.379935 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:46.379949 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:46.381495 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
W20250903 14:27:46.381685 29563 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:46.377429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:49491 (local address 127.19.65.190:44265)
0903 14:27:46.377565 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:46.377570 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:46.377690 (+ 120us) server_negotiation.cc:408] Connection header received
0903 14:27:46.377836 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:46.377839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:46.377888 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:46.377941 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:46.378427 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.378834 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:46.379480 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:46.379665 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:46.381603 (+ 1938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:49491: BlockingRecv error: recv got EOF from 127.19.65.131:49491 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:47.096663 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:47.092254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:44363)
0903 14:27:47.092507 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:47.092520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:47.092615 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:47.092853 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:47.092856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:47.093051 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:47.093236 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.093242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.094073 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.094078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:47.094434 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.094441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.094532 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.094990 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:47.095001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:47.096557 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20250903 14:27:47.096846 29567 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:47.092363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44363 (local address 127.19.65.190:44265)
0903 14:27:47.092590 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:47.092595 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:47.092606 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:47.092706 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:47.092709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:47.092753 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:47.092822 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:47.093348 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.093963 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.094518 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.094639 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.096679 (+ 2040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44363: BlockingRecv error: recv got EOF from 127.19.65.129:44363 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:47.271149 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:47.266701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:45331)
0903 14:27:47.267048 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:47.267060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:47.267163 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:47.267410 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:47.267414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:47.267606 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:47.267804 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.267810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.268482 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.268485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:47.268814 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.268820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.268910 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.269512 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:47.269527 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:47.271013 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":138,"threads_started":1}
W20250903 14:27:47.271229 29567 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:47.266770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:45331 (local address 127.19.65.190:44265)
0903 14:27:47.266958 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:47.266963 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:47.267130 (+ 167us) server_negotiation.cc:408] Connection header received
0903 14:27:47.267249 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:47.267254 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:47.267305 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:47.267375 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:47.267915 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.268387 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.268901 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.269100 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.271132 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:45331: BlockingRecv error: recv got EOF from 127.19.65.130:45331 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:47 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:47.387028 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:47.382274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:52809)
0903 14:27:47.382516 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:47.382531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:47.382638 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:47.382842 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:47.382844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:47.383020 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:47.383201 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.383206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.383929 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.383933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:47.384431 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:47.384440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.384548 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.385310 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:47.385323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:47.386911 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
W20250903 14:27:47.387120 29567 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:47.382415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:52809 (local address 127.19.65.190:44265)
0903 14:27:47.382535 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:47.382539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:47.382594 (+ 55us) server_negotiation.cc:408] Connection header received
0903 14:27:47.382712 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:47.382715 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:47.382759 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:47.382826 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:47.383297 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.383796 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:47.384542 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:47.384771 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:47.387025 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:52809: BlockingRecv error: recv got EOF from 127.19.65.131:52809 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:48.101835 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:48.097367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:48785)
0903 14:27:48.097650 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:48.097664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:48.097755 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:48.097996 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:48.097998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:48.098189 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:48.098365 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.098369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.099119 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.099123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:48.099571 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.099583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.099678 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.100153 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:48.100164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:48.101729 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
W20250903 14:27:48.102017 29571 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:48.097435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:48785 (local address 127.19.65.190:44265)
0903 14:27:48.097695 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:48.097698 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:48.097711 (+ 13us) server_negotiation.cc:408] Connection header received
0903 14:27:48.097853 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:48.097856 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:48.097898 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:48.097973 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:48.098446 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.099018 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.099672 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.099789 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.101863 (+ 2074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:48785: BlockingRecv error: recv got EOF from 127.19.65.129:48785 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:48.275753 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:48.271894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:48487)
0903 14:27:48.272145 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:48.272156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:48.272241 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:48.272436 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:48.272438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:48.272585 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:48.272759 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.272764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.273345 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.273347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:48.273676 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.273681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.273753 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.274245 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:48.274256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:48.275658 (+ 1402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20250903 14:27:48.275861 29571 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:48.272003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:48487 (local address 127.19.65.190:44265)
0903 14:27:48.272126 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:48.272129 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:48.272218 (+ 89us) server_negotiation.cc:408] Connection header received
0903 14:27:48.272309 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:48.272311 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:48.272351 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:48.272413 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:48.272845 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.273274 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.273744 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.273864 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.275767 (+ 1903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:48487: BlockingRecv error: recv got EOF from 127.19.65.130:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:48 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:48.392046 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:48.387669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:43817)
0903 14:27:48.387934 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:48.387947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:48.388067 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:48.388291 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:48.388294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:48.388457 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:48.388660 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.388668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.389336 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.389339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:48.389752 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:48.389759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.389864 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.390364 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:48.390376 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:48.391913 (+ 1537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20250903 14:27:48.392124 29571 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:48.387760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43817 (local address 127.19.65.190:44265)
0903 14:27:48.387902 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:48.387907 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:48.388010 (+ 103us) server_negotiation.cc:408] Connection header received
0903 14:27:48.388162 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:48.388165 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:48.388207 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:48.388265 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:48.388767 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.389245 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:48.389843 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:48.390011 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:48.392019 (+ 2008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:43817: BlockingRecv error: recv got EOF from 127.19.65.131:43817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:49.106810 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:49.102384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:53333)
0903 14:27:49.102632 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:49.102644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:49.102726 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:49.103013 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:49.103015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:49.103228 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:49.103411 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.103417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.104180 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.104185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:49.104549 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.104557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.104645 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.105104 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:49.105115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:49.106709 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
W20250903 14:27:49.106978 29575 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:49.102459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53333 (local address 127.19.65.190:44265)
0903 14:27:49.102681 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:49.102685 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:49.102697 (+ 12us) server_negotiation.cc:408] Connection header received
0903 14:27:49.102826 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:49.102830 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:49.102893 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:49.102977 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:49.103520 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.104060 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.104634 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.104765 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.106831 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:53333: BlockingRecv error: recv got EOF from 127.19.65.129:53333 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:49.289086 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:49.276449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:40603)
0903 14:27:49.276703 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:49.276715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:49.276802 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:49.276989 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:49.276991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:49.277168 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:49.277356 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.277362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.278034 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.278036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:49.278390 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.278396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.278471 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.287398 (+ 8927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:49.287411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:49.288978 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
W20250903 14:27:49.289189 29575 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:49.276602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:40603 (local address 127.19.65.190:44265)
0903 14:27:49.276719 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:49.276722 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:49.276762 (+ 40us) server_negotiation.cc:408] Connection header received
0903 14:27:49.276868 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:49.276870 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:49.276906 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:49.276962 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:49.277452 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.277947 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.278470 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.278579 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.289096 (+ 10517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:40603: BlockingRecv error: recv got EOF from 127.19.65.130:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:49 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:49.396939 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:49.392666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:56493)
0903 14:27:49.392928 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:49.392940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:49.393027 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:49.393216 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:49.393218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:49.393355 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:49.393537 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.393542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.394286 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.394290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:49.394693 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:49.394702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.394815 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.395379 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:49.395397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:49.396837 (+ 1440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
W20250903 14:27:49.397064 29575 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:49.392801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:56493 (local address 127.19.65.190:44265)
0903 14:27:49.392919 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:49.392922 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:49.392994 (+ 72us) server_negotiation.cc:408] Connection header received
0903 14:27:49.393089 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:49.393090 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:49.393137 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:49.393194 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:49.393655 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.394184 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:49.394804 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:49.394988 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:49.396953 (+ 1965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:56493: BlockingRecv error: recv got EOF from 127.19.65.131:56493 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:50.111824 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:50.107541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:33849)
0903 14:27:50.107794 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:50.107806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:50.107910 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:50.108202 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:50.108206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:50.108450 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:50.108641 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.108646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.109297 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.109299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:50.109653 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.109659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.109736 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.110212 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:50.110224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:50.111727 (+ 1503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
W20250903 14:27:50.112013 29579 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:50.107580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33849 (local address 127.19.65.190:44265)
0903 14:27:50.107819 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:50.107823 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:50.107866 (+ 43us) server_negotiation.cc:408] Connection header received
0903 14:27:50.108004 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:50.108007 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:50.108067 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:50.108165 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:50.108742 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.109220 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.109733 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.109853 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.111845 (+ 1992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:33849: BlockingRecv error: recv got EOF from 127.19.65.129:33849 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:50.293677 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:50.289804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:53205)
0903 14:27:50.290040 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:50.290051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:50.290133 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:50.290317 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:50.290319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:50.290464 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:50.290634 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.290639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.291196 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.291198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:50.291507 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.291511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.291581 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.292066 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:50.292077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:50.293578 (+ 1501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
W20250903 14:27:50.293795 29579 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:50.289923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:53205 (local address 127.19.65.190:44265)
0903 14:27:50.290048 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:50.290051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:50.290103 (+ 52us) server_negotiation.cc:408] Connection header received
0903 14:27:50.290198 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:50.290200 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:50.290238 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:50.290299 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:50.290707 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.291122 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.291583 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.291698 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.293683 (+ 1985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:53205: BlockingRecv error: recv got EOF from 127.19.65.130:53205 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:50 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:50.402016 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:50.397664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:52003)
0903 14:27:50.398008 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:50.398020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:50.398125 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:50.398394 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:50.398397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:50.398636 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:50.398812 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.398818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.399504 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.399507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:50.399844 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:50.399849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.399946 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.400409 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:50.400420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:50.401906 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":94,"threads_started":1}
W20250903 14:27:50.402086 29579 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:50.397709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:52003 (local address 127.19.65.190:44265)
0903 14:27:50.397832 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:50.397837 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:50.398093 (+ 256us) server_negotiation.cc:408] Connection header received
0903 14:27:50.398215 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:50.398219 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:50.398266 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:50.398354 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:50.398928 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.399369 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:50.399948 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:50.400131 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:50.402008 (+ 1877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:52003: BlockingRecv error: recv got EOF from 127.19.65.131:52003 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:51.116811 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:51.112550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:59353)
0903 14:27:51.112873 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:51.112887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:51.113002 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:51.113247 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:51.113249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:51.113449 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:51.113656 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.113662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.114314 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.114316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:51.114653 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.114658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.114733 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.115213 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:51.115223 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:51.116709 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
W20250903 14:27:51.116921 29583 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:51.112603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:59353 (local address 127.19.65.190:44265)
0903 14:27:51.112870 (+ 267us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:51.112876 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:51.112949 (+ 73us) server_negotiation.cc:408] Connection header received
0903 14:27:51.113091 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:51.113094 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:51.113142 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:51.113218 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:51.113742 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.114232 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.114732 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.114850 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.116821 (+ 1971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:59353: BlockingRecv error: recv got EOF from 127.19.65.129:59353 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":64,"threads_started":1}
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:51.298775 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:51.294296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:52519)
0903 14:27:51.294583 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:51.294596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:51.294690 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:51.294884 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:51.294887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:51.295083 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:51.295268 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.295274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.296072 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.296075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:51.296417 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.296423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.296506 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.297003 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:51.297020 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:51.298678 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":70,"threads_started":1}
W20250903 14:27:51.298882 29583 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:51.294367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52519 (local address 127.19.65.190:44265)
0903 14:27:51.294533 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:51.294537 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:51.294655 (+ 118us) server_negotiation.cc:408] Connection header received
0903 14:27:51.294756 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:51.294758 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:51.294800 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:51.294862 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:51.295464 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.295976 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.296502 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.296628 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.298792 (+ 2164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:52519: BlockingRecv error: recv got EOF from 127.19.65.130:52519 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:51 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:51.406823 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:51.402631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.131:57303)
0903 14:27:51.402912 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:51.402925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:51.403037 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:51.403313 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:51.403316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:51.403523 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:51.403717 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.403724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.404344 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.404346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:51.404679 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:51.404685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.404766 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.405242 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:51.405254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:51.406725 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20250903 14:27:51.406922 29583 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:51.402661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57303 (local address 127.19.65.190:44265)
0903 14:27:51.402821 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:51.402825 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:51.403011 (+ 186us) server_negotiation.cc:408] Connection header received
0903 14:27:51.403132 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:51.403135 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:51.403182 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:51.403266 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:51.403816 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.404258 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:51.404769 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:51.404881 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:51.406832 (+ 1951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:57303: BlockingRecv error: recv got EOF from 127.19.65.131:57303 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 03 14:27:52 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:52 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:52.122128 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.117709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.129:53739)
0903 14:27:52.117987 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:52.118003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:52.118102 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:52.118398 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:52.118401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:52.118645 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:52.118893 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.118903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.119600 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.119603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:52.119923 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.119928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.120006 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.120477 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:52.120490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:52.122031 (+ 1541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
W20250903 14:27:52.122236 29587 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:52.117786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53739 (local address 127.19.65.190:44265)
0903 14:27:52.118080 (+ 294us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.118085 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.118101 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:27:52.118206 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.118210 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.118263 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.118362 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.119012 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.119514 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.120015 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.120130 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.122140 (+ 2010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:53739: BlockingRecv error: recv got EOF from 127.19.65.129:53739 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
Sep 03 14:27:52 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Sep 03 14:27:52 dist-test-slave-4wpb krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20250903 14:27:52.303748 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.299489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44265 (local address 127.19.65.130:41785)
0903 14:27:52.299706 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:52.299718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:52.299801 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:52.300053 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:52.300055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:52.300195 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:52.300390 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.300395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.301224 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.301226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:52.301663 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.301670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.301750 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.302194 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:52.302204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:52.303635 (+ 1431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:44265: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
W20250903 14:27:52.303856 29587 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:52.299537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41785 (local address 127.19.65.190:44265)
0903 14:27:52.299650 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.299654 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.299780 (+ 126us) server_negotiation.cc:408] Connection header received
0903 14:27:52.299912 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.299915 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.299957 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.300030 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.300485 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.301135 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.301751 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.301872 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.303764 (+ 1892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:41785: BlockingRecv error: recv got EOF from 127.19.65.130:41785 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:52.330912 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29112
I20250903 14:27:52.335942 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29263
I20250903 14:27:52.340831 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29397
I20250903 14:27:52.345590 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29042
2025-09-03T14:27:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18343 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-09-03T14:27:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:52Z Disabled control of system clock
I20250903 14:27:52.372226 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37681
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250903 14:27:52.463999 29592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:52.464149 29592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:52.464184 29592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:52.466799 29592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:52.466841 29592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:52.466854 29592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:52.466871 29592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:52.466889 29592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:52.470039 29592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:37681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:52.470741 29592 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:52.471558 29592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:52.476155 29600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.476251 29592 server_base.cc:1047] running on GCE node
W20250903 14:27:52.476254 29597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:52.476171 29598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.476752 29592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:52.477515 29592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:52.478657 29592 hybrid_clock.cc:648] HybridClock initialized: now 1756909672478642 us; error 30 us; skew 500 ppm
I20250903 14:27:52.480010 29592 webserver.cc:480] Webserver started at http://127.19.65.190:38811/ using document root <none> and password file <none>
I20250903 14:27:52.480458 29592 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:52.480505 29592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:52.480671 29592 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:52.482214 29592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "47544cefc4b1418680c82d92670fdf9a"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "bbc8a6491897f1dc3d119a340449097e"
server_key_iv: "85dc76df3150f7c75cf3f57000c41587"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.482648 29592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "47544cefc4b1418680c82d92670fdf9a"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "bbc8a6491897f1dc3d119a340449097e"
server_key_iv: "85dc76df3150f7c75cf3f57000c41587"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.485332 29592 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 14:27:52.487077 29606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.488160 29592 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:52.488262 29592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "47544cefc4b1418680c82d92670fdf9a"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "bbc8a6491897f1dc3d119a340449097e"
server_key_iv: "85dc76df3150f7c75cf3f57000c41587"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.488343 29592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:52.500972 29592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:52.501439 29592 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:52.501611 29592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:52.507571 29592 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:37681
I20250903 14:27:52.507581 29658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:37681 every 8 connection(s)
I20250903 14:27:52.508378 29592 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:52.510550 29659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:52.515105 29659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a: Bootstrap starting.
I20250903 14:27:52.516006 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29592
I20250903 14:27:52.516108 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:52.516309 19718 external_mini_cluster.cc:1468] Setting key 91e28c6332bddbf6173bb01e2e632354
I20250903 14:27:52.517133 29659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:52.517850 29659 log.cc:826] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:52.519520 29659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a: No bootstrap required, opened a new log
I20250903 14:27:52.522148 29659 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } }
I20250903 14:27:52.522379 29659 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:52.522454 29659 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47544cefc4b1418680c82d92670fdf9a, State: Initialized, Role: FOLLOWER
I20250903 14:27:52.522847 29659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } }
I20250903 14:27:52.522933 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.517190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.19.65.190:37681)
0903 14:27:52.517757 (+ 567us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.517764 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.517793 (+ 29us) server_negotiation.cc:408] Connection header received
0903 14:27:52.518301 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.518317 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.518565 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.518807 (+ 242us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.519296 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.519959 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.520722 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.520952 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.521551 (+ 599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:52.521567 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:52.521576 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:52.521907 (+ 331us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:52.521940 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:52.521953 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:52.522028 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:52.522214 (+ 186us) server_negotiation.cc:300] Negotiation successful
0903 14:27:52.522427 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20250903 14:27:52.522956 29659 raft_consensus.cc:397] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:52.523017 29659 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:52.523140 29659 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:52.523764 29659 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } }
I20250903 14:27:52.524039 29659 leader_election.cc:304] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47544cefc4b1418680c82d92670fdf9a; no voters:
I20250903 14:27:52.524252 29659 leader_election.cc:290] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:52.524340 29664 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:52.524780 29664 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [term 1 LEADER]: Becoming Leader. State: Replica: 47544cefc4b1418680c82d92670fdf9a, State: Running, Role: LEADER
I20250903 14:27:52.525051 29664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } }
I20250903 14:27:52.525152 29659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:52.527854 29666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47544cefc4b1418680c82d92670fdf9a. Latest consensus state: current_term: 1 leader_uuid: "47544cefc4b1418680c82d92670fdf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } } }
I20250903 14:27:52.528043 29666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:52.528139 29665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47544cefc4b1418680c82d92670fdf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47544cefc4b1418680c82d92670fdf9a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37681 } } }
I20250903 14:27:52.528232 29665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:52.528493 29679 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:52.530602 29679 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:52.534477 29679 catalog_manager.cc:1349] Generated new cluster ID: cd055801d5bb4d6ab20a0546340adb1f
I20250903 14:27:52.534533 29679 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:52.546869 29679 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:52.547560 29679 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:52.552248 29679 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 47544cefc4b1418680c82d92670fdf9a: Generated new TSK 0
I20250903 14:27:52.552724 29679 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:27:52.555321 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--builtin_ntp_servers=127.19.65.148:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20250903 14:27:52.646939 29683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:52.647112 29683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:52.647181 29683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:52.650101 29683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:52.650143 29683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:52.650235 29683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:52.653630 29683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:52.654471 29683 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:52.655382 29683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:52.660725 29689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:52.660817 29688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.661003 29683 server_base.cc:1047] running on GCE node
W20250903 14:27:52.660729 29691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.661392 29683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:52.661849 29683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:52.662998 29683 hybrid_clock.cc:648] HybridClock initialized: now 1756909672662978 us; error 37 us; skew 500 ppm
I20250903 14:27:52.664381 29683 webserver.cc:480] Webserver started at http://127.19.65.129:34431/ using document root <none> and password file <none>
I20250903 14:27:52.664812 29683 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:52.664857 29683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:52.665009 29683 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:52.666440 29683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "3317282961d14980a5ded9f5b4935c9e"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "4be38a431416708703247511c34695f7"
server_key_iv: "808e7d1d1ad7ebd4f181a2441c93d0b7"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.666828 29683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "3317282961d14980a5ded9f5b4935c9e"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "4be38a431416708703247511c34695f7"
server_key_iv: "808e7d1d1ad7ebd4f181a2441c93d0b7"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.669664 29683 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:27:52.671336 29697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.672113 29683 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:52.672191 29683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "3317282961d14980a5ded9f5b4935c9e"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "4be38a431416708703247511c34695f7"
server_key_iv: "808e7d1d1ad7ebd4f181a2441c93d0b7"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.672273 29683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:52.683553 29683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:52.684000 29683 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:52.684135 29683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:52.684543 29683 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:52.685263 29683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:52.685305 29683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.685341 29683 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:52.685366 29683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.693022 29683 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:41967
I20250903 14:27:52.693060 29810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:41967 every 8 connection(s)
I20250903 14:27:52.693838 29683 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:52.699750 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29683
I20250903 14:27:52.699872 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:52.700131 19718 external_mini_cluster.cc:1468] Setting key 61c9a0693e3c5aad290e5f3be96cbfdd
I20250903 14:27:52.700325 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.695258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44547 (local address 127.19.65.190:37681)
0903 14:27:52.695367 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.695370 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.696041 (+ 671us) server_negotiation.cc:408] Connection header received
0903 14:27:52.696697 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.696700 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.696741 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.696803 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.697631 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.698101 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.698657 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.698782 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.699830 (+ 1048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:52.699847 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:52.699851 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:52.699864 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:52.699882 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:52.699890 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:52.699958 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:52.700156 (+ 198us) server_negotiation.cc:300] Negotiation successful
0903 14:27:52.700214 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:52.700908 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.695451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37681 (local address 127.19.65.129:44547)
0903 14:27:52.695931 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:52.695958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:52.696562 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:52.696911 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:52.696918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:52.696978 (+ 60us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:52.697507 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.697531 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.698198 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.698200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:52.698579 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.698585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.698726 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.699645 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:52.699646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:52.699669 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:52.699712 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:52.699970 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:52.699979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:52.699985 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 14:27:52.700160 (+ 175us) client_negotiation.cc:240] Negotiation successful
0903 14:27:52.700353 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20250903 14:27:52.701908 29811 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37681
I20250903 14:27:52.701908 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--builtin_ntp_servers=127.19.65.148:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250903 14:27:52.702118 29811 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:52.702522 29811 heartbeater.cc:507] Master 127.19.65.190:37681 requested a full tablet report, sending...
I20250903 14:27:52.703752 29623 ts_manager.cc:194] Registered new tserver with Master: 3317282961d14980a5ded9f5b4935c9e (127.19.65.129:41967)
I20250903 14:27:52.704691 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.129:44547
W20250903 14:27:52.791590 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:52.791734 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:52.791775 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:52.794533 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:52.794574 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:52.794648 29814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:52.797812 29814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:52.798614 29814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:52.799496 29814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:52.804375 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:52.804378 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.804563 29814 server_base.cc:1047] running on GCE node
W20250903 14:27:52.804600 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.804890 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:52.805323 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:52.806453 29814 hybrid_clock.cc:648] HybridClock initialized: now 1756909672806439 us; error 27 us; skew 500 ppm
I20250903 14:27:52.807849 29814 webserver.cc:480] Webserver started at http://127.19.65.130:45145/ using document root <none> and password file <none>
I20250903 14:27:52.808329 29814 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:52.808388 29814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:52.808543 29814 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:52.810055 29814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "4b2691875c0c47539f83f6f7aca5641b"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "0c6c7d4b2d77ac04bfc25e72dbf94f99"
server_key_iv: "273b6fdfb641a8582b30a6a4c4cbab21"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.810518 29814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "4b2691875c0c47539f83f6f7aca5641b"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "0c6c7d4b2d77ac04bfc25e72dbf94f99"
server_key_iv: "273b6fdfb641a8582b30a6a4c4cbab21"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.813383 29814 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:27:52.815217 29828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.816005 29814 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:52.816107 29814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "4b2691875c0c47539f83f6f7aca5641b"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "0c6c7d4b2d77ac04bfc25e72dbf94f99"
server_key_iv: "273b6fdfb641a8582b30a6a4c4cbab21"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.816181 29814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:52.825277 29814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:52.825769 29814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:52.825903 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:52.826318 29814 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:52.827083 29814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:52.827124 29814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.827195 29814 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:52.827219 29814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.834800 29814 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45691
I20250903 14:27:52.834828 29941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:45691 every 8 connection(s)
I20250903 14:27:52.835510 29814 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:52.837893 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29814
I20250903 14:27:52.838006 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:52.838227 19718 external_mini_cluster.cc:1468] Setting key 26465761075d862e95e87458f1d365b3
I20250903 14:27:52.840016 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--builtin_ntp_servers=127.19.65.148:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250903 14:27:52.842478 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.836816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57875 (local address 127.19.65.190:37681)
0903 14:27:52.836954 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.836959 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.837730 (+ 771us) server_negotiation.cc:408] Connection header received
0903 14:27:52.838514 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.838518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.838578 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.838667 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.839497 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.840178 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.840729 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.840946 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.841954 (+ 1008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:52.841975 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:52.841980 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:52.841995 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:52.842018 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:52.842029 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:52.842112 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:52.842222 (+ 110us) server_negotiation.cc:300] Negotiation successful
0903 14:27:52.842289 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:27:52.842911 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.837050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37681 (local address 127.19.65.130:57875)
0903 14:27:52.837612 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:52.837643 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:52.838335 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:52.838820 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:52.838827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:52.838889 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:52.839362 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.839377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.840276 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.840279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:52.840639 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.840645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.840769 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.841757 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:52.841759 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:52.841784 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:52.841828 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:52.842115 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:52.842119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:52.842123 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 14:27:52.842210 (+ 87us) client_negotiation.cc:240] Negotiation successful
0903 14:27:52.842364 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20250903 14:27:52.843801 29942 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37681
I20250903 14:27:52.843979 29942 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:52.844384 29942 heartbeater.cc:507] Master 127.19.65.190:37681 requested a full tablet report, sending...
I20250903 14:27:52.845209 29623 ts_manager.cc:194] Registered new tserver with Master: 4b2691875c0c47539f83f6f7aca5641b (127.19.65.130:45691)
I20250903 14:27:52.845952 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.130:57875
W20250903 14:27:52.930881 29945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:52.931026 29945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:52.931063 29945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:52.933737 29945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:52.933773 29945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:52.933830 29945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:52.937079 29945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:52.937867 29945 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:52.938743 29945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:52.943598 29951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:52.943656 29953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:52.943606 29950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:52.943843 29945 server_base.cc:1047] running on GCE node
I20250903 14:27:52.944211 29945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:52.944639 29945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:52.945791 29945 hybrid_clock.cc:648] HybridClock initialized: now 1756909672945779 us; error 32 us; skew 500 ppm
I20250903 14:27:52.947149 29945 webserver.cc:480] Webserver started at http://127.19.65.131:36431/ using document root <none> and password file <none>
I20250903 14:27:52.947635 29945 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:52.947693 29945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:52.947849 29945 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:52.949251 29945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "f16246a4c99c476a9c352a11b49a35d4"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "35788c78bd362d42008c33a20f7ccafa"
server_key_iv: "732b47b19c9202a4dea8b6d3690838f4"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.949685 29945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "f16246a4c99c476a9c352a11b49a35d4"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "35788c78bd362d42008c33a20f7ccafa"
server_key_iv: "732b47b19c9202a4dea8b6d3690838f4"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.952246 29945 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:52.953974 29959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.954802 29945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:52.954893 29945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "f16246a4c99c476a9c352a11b49a35d4"
format_stamp: "Formatted at 2025-09-03 14:27:52 on dist-test-slave-4wpb"
server_key: "35788c78bd362d42008c33a20f7ccafa"
server_key_iv: "732b47b19c9202a4dea8b6d3690838f4"
server_key_version: "encryptionkey@0"
I20250903 14:27:52.954970 29945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:52.968741 29945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:52.969178 29945 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:52.969316 29945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:52.969760 29945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:52.970476 29945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:52.970520 29945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.970569 29945 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:52.970602 29945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:52.978739 29945 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:36985
I20250903 14:27:52.978765 30072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:36985 every 8 connection(s)
I20250903 14:27:52.979507 29945 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:52.984846 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 29945
I20250903 14:27:52.984974 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:52.985316 19718 external_mini_cluster.cc:1468] Setting key 1f52a652971c07682aa619882556e0d0
I20250903 14:27:52.986169 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.980901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46763 (local address 127.19.65.190:37681)
0903 14:27:52.981040 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:52.981044 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:52.981751 (+ 707us) server_negotiation.cc:408] Connection header received
0903 14:27:52.982419 (+ 668us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:52.982422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:52.982467 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:52.982534 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:52.983302 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.983732 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.984290 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.984407 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.985690 (+ 1283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:52.985710 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:52.985716 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:52.985732 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:52.985755 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:52.985765 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:52.985825 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:52.985987 (+ 162us) server_negotiation.cc:300] Negotiation successful
0903 14:27:52.986042 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 14:27:52.986709 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:52.981110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:37681 (local address 127.19.65.131:46763)
0903 14:27:52.981645 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:52.981672 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:52.982278 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:52.982643 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:52.982649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:52.982712 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:52.983204 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.983214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.983827 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:52.983830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:52.984211 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:52.984218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:52.984344 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:52.985437 (+ 1093us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:52.985439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:52.985491 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:52.985544 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:52.985844 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:52.985851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:52.985856 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 14:27:52.985972 (+ 116us) client_negotiation.cc:240] Negotiation successful
0903 14:27:52.986161 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20250903 14:27:52.987552 30073 heartbeater.cc:344] Connected to a master server at 127.19.65.190:37681
I20250903 14:27:52.987733 30073 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:52.988158 30073 heartbeater.cc:507] Master 127.19.65.190:37681 requested a full tablet report, sending...
I20250903 14:27:52.988919 29623 ts_manager.cc:194] Registered new tserver with Master: f16246a4c99c476a9c352a11b49a35d4 (127.19.65.131:36985)
I20250903 14:27:52.989365 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.131:46763
I20250903 14:27:52.998703 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 14:27:53.001111 29662 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:53.000336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.19.65.190:37681)
0903 14:27:53.000463 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.000467 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.000637 (+ 170us) server_negotiation.cc:408] Connection header received
0903 14:27:53.000740 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.000743 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.000780 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.000843 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.001027 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45600: BlockingRecv error: recv got EOF from 127.0.0.1:45600 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 14:27:53.001368 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29683
I20250903 14:27:53.006940 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29814
I20250903 14:27:53.012024 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29945
I20250903 14:27:53.016515 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 29592
2025-09-03T14:27:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (670 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-09-03T14:27:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:53Z Disabled control of system clock
I20250903 14:27:53.042220 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43349
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250903 14:27:53.133690 30084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:53.133839 30084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:53.133878 30084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:53.136651 30084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:53.136695 30084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:53.136708 30084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:53.136724 30084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:53.136744 30084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:53.140264 30084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:53.140990 30084 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:53.141944 30084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:53.146399 30089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.146399 30092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.146396 30090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:53.146517 30084 server_base.cc:1047] running on GCE node
I20250903 14:27:53.146925 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:53.147670 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:53.148818 30084 hybrid_clock.cc:648] HybridClock initialized: now 1756909673148797 us; error 34 us; skew 500 ppm
I20250903 14:27:53.150153 30084 webserver.cc:480] Webserver started at http://127.19.65.190:32797/ using document root <none> and password file <none>
I20250903 14:27:53.150632 30084 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:53.150681 30084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:53.150828 30084 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:53.152190 30084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "0e6b44d158f1455d9809a5127aa608a9"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d365907097b9583b1c9d5fc9769c94bd"
server_key_iv: "075e06db19ed4fcaf80e353788bfab3c"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.152565 30084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "0e6b44d158f1455d9809a5127aa608a9"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d365907097b9583b1c9d5fc9769c94bd"
server_key_iv: "075e06db19ed4fcaf80e353788bfab3c"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.155256 30084 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250903 14:27:53.157146 30098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.158027 30084 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:53.158138 30084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "0e6b44d158f1455d9809a5127aa608a9"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d365907097b9583b1c9d5fc9769c94bd"
server_key_iv: "075e06db19ed4fcaf80e353788bfab3c"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.158219 30084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:53.161968 30084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:53.162443 30084 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:53.162582 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:53.168255 30084 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43349
I20250903 14:27:53.168258 30150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:43349 every 8 connection(s)
I20250903 14:27:53.169042 30084 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:53.171025 30151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:53.175272 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9: Bootstrap starting.
I20250903 14:27:53.175761 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30084
I20250903 14:27:53.175863 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:53.176051 19718 external_mini_cluster.cc:1468] Setting key f94fba5abd93721136b775e35cb6be97
I20250903 14:27:53.177218 30151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:53.177995 30151 log.cc:826] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:53.179777 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9: No bootstrap required, opened a new log
I20250903 14:27:53.180426 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.176864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.19.65.190:43349)
0903 14:27:53.177518 (+ 654us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.177528 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.177564 (+ 36us) server_negotiation.cc:408] Connection header received
0903 14:27:53.178176 (+ 612us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.178197 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.178509 (+ 312us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.178801 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.179426 (+ 625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:53.179431 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:53.179441 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:53.179451 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:53.179492 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:53.179506 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:53.179618 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:53.179703 (+ 85us) server_negotiation.cc:300] Negotiation successful
0903 14:27:53.179891 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20250903 14:27:53.182088 30151 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } }
I20250903 14:27:53.182255 30151 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:53.182302 30151 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6b44d158f1455d9809a5127aa608a9, State: Initialized, Role: FOLLOWER
I20250903 14:27:53.182595 30151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } }
I20250903 14:27:53.182683 30151 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:53.182720 30151 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:53.182787 30151 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:53.183436 30151 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } }
I20250903 14:27:53.183670 30151 leader_election.cc:304] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e6b44d158f1455d9809a5127aa608a9; no voters:
I20250903 14:27:53.183866 30151 leader_election.cc:290] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:53.183949 30156 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:53.184132 30156 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e6b44d158f1455d9809a5127aa608a9, State: Running, Role: LEADER
I20250903 14:27:53.184398 30156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } }
I20250903 14:27:53.184708 30151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:53.185719 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e6b44d158f1455d9809a5127aa608a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } } }
I20250903 14:27:53.185832 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:53.185719 30158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e6b44d158f1455d9809a5127aa608a9. Latest consensus state: current_term: 1 leader_uuid: "0e6b44d158f1455d9809a5127aa608a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6b44d158f1455d9809a5127aa608a9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43349 } } }
I20250903 14:27:53.186013 30158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9 [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:53.186234 30165 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:53.188583 30165 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:53.192997 30165 catalog_manager.cc:1349] Generated new cluster ID: 2753c9d0125f4461b6d0a52a58101840
I20250903 14:27:53.193051 30165 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:53.210345 30165 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:53.210552 30165 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:53.217496 30165 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0e6b44d158f1455d9809a5127aa608a9: Generated new TSK 0
I20250903 14:27:53.218036 30165 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:27:53.223893 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--builtin_ntp_servers=127.19.65.148:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250903 14:27:53.314271 30175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:53.314467 30175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:53.314541 30175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:53.317320 30175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:53.317359 30175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:53.317427 30175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:53.320660 30175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:53.321431 30175 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:53.322351 30175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:53.327538 30180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.327600 30183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.327548 30181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:53.327785 30175 server_base.cc:1047] running on GCE node
I20250903 14:27:53.328173 30175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:53.328601 30175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:53.329758 30175 hybrid_clock.cc:648] HybridClock initialized: now 1756909673329723 us; error 31 us; skew 500 ppm
I20250903 14:27:53.331125 30175 webserver.cc:480] Webserver started at http://127.19.65.129:42707/ using document root <none> and password file <none>
I20250903 14:27:53.331568 30175 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:53.331614 30175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:53.331769 30175 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:53.333110 30175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "8f2853140c3940cab8a2ec6496accd93"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "fb9050c65347605d6635b1a553e0bd38"
server_key_iv: "4c99e31c37156e11606d7ee1c5885061"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.333513 30175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "8f2853140c3940cab8a2ec6496accd93"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "fb9050c65347605d6635b1a553e0bd38"
server_key_iv: "4c99e31c37156e11606d7ee1c5885061"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.336228 30175 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 14:27:53.338011 30189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.338881 30175 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 14:27:53.338966 30175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "8f2853140c3940cab8a2ec6496accd93"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "fb9050c65347605d6635b1a553e0bd38"
server_key_iv: "4c99e31c37156e11606d7ee1c5885061"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.339036 30175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:53.342729 30175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:53.343159 30175 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:53.343286 30175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:53.343678 30175 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:53.344367 30175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:53.344472 30175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.344560 30175 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:53.344594 30175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.353060 30175 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:45393
I20250903 14:27:53.353075 30302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:45393 every 8 connection(s)
I20250903 14:27:53.353838 30175 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:53.358042 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.355227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:54773 (local address 127.19.65.190:43349)
0903 14:27:53.355338 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.355341 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.356024 (+ 683us) server_negotiation.cc:408] Connection header received
0903 14:27:53.356638 (+ 614us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.356642 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.356684 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.356754 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.357625 (+ 871us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:53.357628 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:53.357631 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:53.357642 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:53.357664 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:53.357671 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:53.357750 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:53.357877 (+ 127us) server_negotiation.cc:300] Negotiation successful
0903 14:27:53.357917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:53.358280 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30175
I20250903 14:27:53.358404 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:53.358628 19718 external_mini_cluster.cc:1468] Setting key d1ba7aec796d4a774c1f9b8f79ca9712
I20250903 14:27:53.358634 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.355434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43349 (local address 127.19.65.129:54773)
0903 14:27:53.355922 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:53.355948 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:53.356482 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:53.356874 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:53.356880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:53.356942 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:53.357407 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:53.357408 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:53.357431 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:53.357486 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:53.357744 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:53.357751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:53.357757 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 14:27:53.357909 (+ 152us) client_negotiation.cc:240] Negotiation successful
0903 14:27:53.358084 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20250903 14:27:53.359822 30303 heartbeater.cc:344] Connected to a master server at 127.19.65.190:43349
I20250903 14:27:53.360000 30303 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:53.360297 30303 heartbeater.cc:507] Master 127.19.65.190:43349 requested a full tablet report, sending...
I20250903 14:27:53.360349 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--builtin_ntp_servers=127.19.65.148:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250903 14:27:53.361521 30115 ts_manager.cc:194] Registered new tserver with Master: 8f2853140c3940cab8a2ec6496accd93 (127.19.65.129:45393)
W20250903 14:27:53.449047 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:53.449208 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:53.449278 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:53.451957 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:53.451998 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:53.452112 30306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:53.455287 30306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:53.456046 30306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:53.456954 30306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:53.462016 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.462101 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:53.462087 30306 server_base.cc:1047] running on GCE node
W20250903 14:27:53.462004 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:53.462502 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:53.462930 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:53.464063 30306 hybrid_clock.cc:648] HybridClock initialized: now 1756909673464051 us; error 27 us; skew 500 ppm
I20250903 14:27:53.465364 30306 webserver.cc:480] Webserver started at http://127.19.65.130:42151/ using document root <none> and password file <none>
I20250903 14:27:53.465847 30306 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:53.465901 30306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:53.466054 30306 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:53.467386 30306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "1abc9273144743488bbd131aa92ff9dd"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d21b22c33d0b1eb0d2aa6d9aea84805c"
server_key_iv: "3dbed097403f6df9a0a89aa00457ecd7"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.467842 30306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "1abc9273144743488bbd131aa92ff9dd"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d21b22c33d0b1eb0d2aa6d9aea84805c"
server_key_iv: "3dbed097403f6df9a0a89aa00457ecd7"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.470628 30306 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 14:27:53.472275 30320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.473004 30306 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250903 14:27:53.473104 30306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "1abc9273144743488bbd131aa92ff9dd"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "d21b22c33d0b1eb0d2aa6d9aea84805c"
server_key_iv: "3dbed097403f6df9a0a89aa00457ecd7"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.473168 30306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:53.476900 30306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:53.477344 30306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:53.477500 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:53.477953 30306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:53.478714 30306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:53.478766 30306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.478814 30306 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:53.478842 30306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.486567 30306 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:35421
I20250903 14:27:53.486630 30433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:35421 every 8 connection(s)
I20250903 14:27:53.487282 30306 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:53.491719 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.488665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41075 (local address 127.19.65.190:43349)
0903 14:27:53.488796 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.488800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.489557 (+ 757us) server_negotiation.cc:408] Connection header received
0903 14:27:53.490188 (+ 631us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.490191 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.490236 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.490326 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.491317 (+ 991us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:53.491319 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:53.491321 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:53.491330 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:53.491348 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:53.491354 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:53.491427 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:53.491545 (+ 118us) server_negotiation.cc:300] Negotiation successful
0903 14:27:53.491580 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 14:27:53.492254 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.488879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43349 (local address 127.19.65.130:41075)
0903 14:27:53.489413 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:53.489440 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:53.490013 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:53.490453 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:53.490460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:53.490527 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:53.491118 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:53.491120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:53.491143 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:53.491186 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:53.491422 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:53.491428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:53.491434 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 14:27:53.491575 (+ 141us) client_negotiation.cc:240] Negotiation successful
0903 14:27:53.491758 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
I20250903 14:27:53.493228 30434 heartbeater.cc:344] Connected to a master server at 127.19.65.190:43349
I20250903 14:27:53.493407 30434 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:53.493716 30434 heartbeater.cc:507] Master 127.19.65.190:43349 requested a full tablet report, sending...
I20250903 14:27:53.494510 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30306
I20250903 14:27:53.494529 30115 ts_manager.cc:194] Registered new tserver with Master: 1abc9273144743488bbd131aa92ff9dd (127.19.65.130:35421)
I20250903 14:27:53.494621 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:53.494828 19718 external_mini_cluster.cc:1468] Setting key f83108e91721349af88047b0c0aeaa76
I20250903 14:27:53.496344 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--builtin_ntp_servers=127.19.65.148:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250903 14:27:53.586081 30437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:53.586262 30437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:53.586326 30437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:53.589115 30437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:53.589157 30437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:53.589242 30437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:53.592299 30437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:53.593032 30437 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:53.593868 30437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:53.599138 30443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.599164 30442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:53.599197 30445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:53.599248 30437 server_base.cc:1047] running on GCE node
I20250903 14:27:53.599608 30437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:53.600004 30437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:53.601141 30437 hybrid_clock.cc:648] HybridClock initialized: now 1756909673601124 us; error 30 us; skew 500 ppm
I20250903 14:27:53.602526 30437 webserver.cc:480] Webserver started at http://127.19.65.131:38551/ using document root <none> and password file <none>
I20250903 14:27:53.602946 30437 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:53.602993 30437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:53.603179 30437 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:53.604547 30437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "826a38452a014f13bcf7581dd2e3ea2e"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "89f5a8811a522d098b2e549feffde7f1"
server_key_iv: "e4f649c22913a51dbd87520bcb64b8ca"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.604933 30437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "826a38452a014f13bcf7581dd2e3ea2e"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "89f5a8811a522d098b2e549feffde7f1"
server_key_iv: "e4f649c22913a51dbd87520bcb64b8ca"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.607523 30437 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20250903 14:27:53.609320 30451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.610066 30437 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:53.610143 30437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "826a38452a014f13bcf7581dd2e3ea2e"
format_stamp: "Formatted at 2025-09-03 14:27:53 on dist-test-slave-4wpb"
server_key: "89f5a8811a522d098b2e549feffde7f1"
server_key_iv: "e4f649c22913a51dbd87520bcb64b8ca"
server_key_version: "encryptionkey@0"
I20250903 14:27:53.610236 30437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:53.613952 30437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:53.614400 30437 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:53.614535 30437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:53.614950 30437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:53.615657 30437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:53.615696 30437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.615729 30437 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:53.615758 30437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:53.623050 30437 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38445
I20250903 14:27:53.623126 30564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:38445 every 8 connection(s)
I20250903 14:27:53.623731 30437 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:53.627962 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.625067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46265 (local address 127.19.65.190:43349)
0903 14:27:53.625200 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.625203 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.625921 (+ 718us) server_negotiation.cc:408] Connection header received
0903 14:27:53.626565 (+ 644us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.626568 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.626615 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.626684 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.627519 (+ 835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:53.627522 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:53.627525 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 14:27:53.627535 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:53.627575 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 14:27:53.627583 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:53.627663 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:53.627793 (+ 130us) server_negotiation.cc:300] Negotiation successful
0903 14:27:53.627834 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 14:27:53.628516 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:53.625248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43349 (local address 127.19.65.131:46265)
0903 14:27:53.625808 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:53.625835 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:53.626395 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:53.626816 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:53.626822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:53.626884 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:53.627341 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 14:27:53.627342 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:53.627363 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 14:27:53.627403 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:53.627657 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:53.627664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:53.627670 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 14:27:53.627825 (+ 155us) client_negotiation.cc:240] Negotiation successful
0903 14:27:53.627998 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20250903 14:27:53.629712 30565 heartbeater.cc:344] Connected to a master server at 127.19.65.190:43349
I20250903 14:27:53.629894 30565 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:53.630203 30565 heartbeater.cc:507] Master 127.19.65.190:43349 requested a full tablet report, sending...
I20250903 14:27:53.630471 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30437
I20250903 14:27:53.630563 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
I20250903 14:27:53.630779 19718 external_mini_cluster.cc:1468] Setting key a3df82ab30780723a1047eb5c5d7cddb
I20250903 14:27:53.631052 30115 ts_manager.cc:194] Registered new tserver with Master: 826a38452a014f13bcf7581dd2e3ea2e (127.19.65.131:38445)
I20250903 14:27:53.632534 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 14:27:53.634873 30154 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:27:53.634157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.19.65.190:43349)
0903 14:27:53.634270 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:53.634274 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:53.634423 (+ 149us) server_negotiation.cc:408] Connection header received
0903 14:27:53.634530 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:53.634533 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:53.634558 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:53.634616 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:53.634799 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50240: BlockingRecv error: recv got EOF from 127.0.0.1:50240 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 14:27:53.635222 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30175
I20250903 14:27:53.640352 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30306
I20250903 14:27:53.644948 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30437
I20250903 14:27:53.649392 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30084
2025-09-03T14:27:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (632 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:53 dist-test-slave-4wpb krb5kdc[30575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:53 dist-test-slave-4wpb krb5kdc[30575](info): set up 2 sockets
Sep 03 14:27:53 dist-test-slave-4wpb krb5kdc[30575](info): commencing operation
krb5kdc: starting...
W20250903 14:27:55.409943 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.741s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:55 dist-test-slave-4wpb krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:55.517241 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46295
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:55.606269 30591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:55.606429 30591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:55.606490 30591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:55.609231 30591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:27:55.609277 30591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:55.609294 30591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:55.609328 30591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:27:55.609349 30591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:27:55.612629 30591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:55.613399 30591 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:55.614074 30591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:55.618486 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:55.618518 30599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:55.618533 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:55.618726 30591 server_base.cc:1047] running on GCE node
I20250903 14:27:55.619165 30591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:55.619822 30591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:55.620956 30591 hybrid_clock.cc:648] HybridClock initialized: now 1756909675620939 us; error 30 us; skew 500 ppm
Sep 03 14:27:55 dist-test-slave-4wpb krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909675, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:55.623005 30591 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:27:55.623868 30591 webserver.cc:480] Webserver started at http://127.19.65.190:36561/ using document root <none> and password file <none>
I20250903 14:27:55.624279 30591 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:55.624320 30591 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:55.624483 30591 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:55.625933 30591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "b2d8004e32374bf591372cc50a8dc8ee"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "06c1e3fe9caa83312585741805309d1e"
server_key_iv: "17a3d246a63aeadb7adb66cb395276c2"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.626319 30591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "b2d8004e32374bf591372cc50a8dc8ee"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "06c1e3fe9caa83312585741805309d1e"
server_key_iv: "17a3d246a63aeadb7adb66cb395276c2"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.629081 30591 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:27:55.630968 30606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:55.631788 30591 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 14:27:55.631877 30591 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "b2d8004e32374bf591372cc50a8dc8ee"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "06c1e3fe9caa83312585741805309d1e"
server_key_iv: "17a3d246a63aeadb7adb66cb395276c2"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.631946 30591 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:55.656490 30591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:55.667047 30591 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:55.667189 30591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:55.673417 30591 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46295
I20250903 14:27:55.673421 30658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:46295 every 8 connection(s)
I20250903 14:27:55.674232 30591 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:27:55.676257 30659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:55.680672 30659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: Bootstrap starting.
I20250903 14:27:55.681089 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30591
I20250903 14:27:55.681185 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:27:55.681372 19718 external_mini_cluster.cc:1468] Setting key 2cebc9d4b680a91b0faf5e322f1ab734
I20250903 14:27:55.682708 30659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:55.683305 30659 log.cc:826] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:55.684690 30659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: No bootstrap required, opened a new log
I20250903 14:27:55.686740 30659 raft_consensus.cc:357] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } }
I20250903 14:27:55.686880 30659 raft_consensus.cc:383] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:55.686926 30659 raft_consensus.cc:738] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d8004e32374bf591372cc50a8dc8ee, State: Initialized, Role: FOLLOWER
Sep 03 14:27:55 dist-test-slave-4wpb krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:55.687279 30659 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } }
I20250903 14:27:55.687402 30659 raft_consensus.cc:397] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:27:55.687475 30659 raft_consensus.cc:491] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:27:55.687572 30659 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:55.688632 30659 raft_consensus.cc:513] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } }
I20250903 14:27:55.689031 30659 leader_election.cc:304] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d8004e32374bf591372cc50a8dc8ee; no voters:
I20250903 14:27:55.689280 30659 leader_election.cc:290] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:27:55.689615 30664 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:27:55.689793 30664 raft_consensus.cc:695] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [term 1 LEADER]: Becoming Leader. State: Replica: b2d8004e32374bf591372cc50a8dc8ee, State: Running, Role: LEADER
I20250903 14:27:55.690057 30664 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } }
I20250903 14:27:55.690325 30659 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:27:55.691040 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d8004e32374bf591372cc50a8dc8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } } }
I20250903 14:27:55.691226 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [sys.catalog]: This master's current role is: LEADER
I20250903 14:27:55.694183 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d8004e32374bf591372cc50a8dc8ee. Latest consensus state: current_term: 1 leader_uuid: "b2d8004e32374bf591372cc50a8dc8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8004e32374bf591372cc50a8dc8ee" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46295 } } }
I20250903 14:27:55.693403 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:55.682276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.19.65.190:46295)
0903 14:27:55.682615 (+ 339us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:55.682622 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:55.682648 (+ 26us) server_negotiation.cc:408] Connection header received
0903 14:27:55.683056 (+ 408us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:55.683070 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:55.683310 (+ 240us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:55.683590 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:55.684353 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.684985 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:55.685575 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.685806 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:55.687711 (+ 1905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:55.687731 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:55.687743 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:55.687787 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:55.690383 (+ 2596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:55.691046 (+ 663us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:55.691079 (+ 33us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:55.691085 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:55.691203 (+ 118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:55.691550 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:55.691568 (+ 18us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:55.691571 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:55.691925 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:55.692089 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:55.692496 (+ 407us) server_negotiation.cc:300] Negotiation successful
0903 14:27:55.692776 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20250903 14:27:55.694573 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee [sys.catalog]: This master's current role is: LEADER
W20250903 14:27:55.694625 30679 catalog_manager.cc:1560] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:27:55.694756 30679 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:27:55.694958 30673 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:27:55.695536 30673 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:27:55.699949 30673 catalog_manager.cc:1349] Generated new cluster ID: 1da407c27bbb4c5ea92268f67d8f758c
I20250903 14:27:55.700004 30673 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:27:55.713989 30673 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:27:55.714947 30673 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:27:55.723107 30673 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: Generated new TSK 0
I20250903 14:27:55.723690 30673 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:55.761509 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46295
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:55.851263 30687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:55.851434 30687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:55.851481 30687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:55.854135 30687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:55.854212 30687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:55.854297 30687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:27:55.857650 30687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:55.858517 30687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:55.859223 30687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:55.864252 30695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:55.864257 30692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:55.864311 30693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:55.864415 30687 server_base.cc:1047] running on GCE node
I20250903 14:27:55.864706 30687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:55.865073 30687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:55.866220 30687 hybrid_clock.cc:648] HybridClock initialized: now 1756909675866211 us; error 27 us; skew 500 ppm
Sep 03 14:27:55 dist-test-slave-4wpb krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909675, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:55.868331 30687 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:27:55.869225 30687 webserver.cc:480] Webserver started at http://127.19.65.129:39739/ using document root <none> and password file <none>
I20250903 14:27:55.869702 30687 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:55.869752 30687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:55.869902 30687 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:55.871279 30687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "b26d755c2aee4762a3b27bc5c3d6fc65"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "34bcff107239d45d116d1a58672d9330"
server_key_iv: "a670fdfab46ffa80612e1ca3012cec59"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.871677 30687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "b26d755c2aee4762a3b27bc5c3d6fc65"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "34bcff107239d45d116d1a58672d9330"
server_key_iv: "a670fdfab46ffa80612e1ca3012cec59"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.874562 30687 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:27:55.876297 30702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:55.877100 30687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:55.877187 30687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "b26d755c2aee4762a3b27bc5c3d6fc65"
format_stamp: "Formatted at 2025-09-03 14:27:55 on dist-test-slave-4wpb"
server_key: "34bcff107239d45d116d1a58672d9330"
server_key_iv: "a670fdfab46ffa80612e1ca3012cec59"
server_key_version: "encryptionkey@0"
I20250903 14:27:55.877269 30687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:55.890882 30687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:55.893193 30687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:55.893311 30687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:55.893726 30687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:55.894351 30687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:55.894392 30687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:55.894430 30687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:55.894450 30687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:55.902001 30687 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:43135
I20250903 14:27:55.902027 30815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:43135 every 8 connection(s)
I20250903 14:27:55.902683 30687 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:27:55.905156 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30687
I20250903 14:27:55.905259 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:27:55.905522 19718 external_mini_cluster.cc:1468] Setting key 1e96d53a5813fe773b4730724d07b91a
Sep 03 14:27:55 dist-test-slave-4wpb krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909675, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:55.913297 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:55.903969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53137 (local address 127.19.65.190:46295)
0903 14:27:55.904078 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:55.904082 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:55.904604 (+ 522us) server_negotiation.cc:408] Connection header received
0903 14:27:55.905264 (+ 660us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:55.905267 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:55.905322 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:55.905389 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:55.906921 (+ 1532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.907400 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:55.907977 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.908164 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:55.910303 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:55.910317 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:55.910319 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:55.910341 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:55.911615 (+ 1274us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:55.912121 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:55.912123 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:55.912124 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:55.912156 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:55.912429 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:55.912431 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:55.912432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:55.912544 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:55.912606 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:55.913098 (+ 492us) server_negotiation.cc:300] Negotiation successful
0903 14:27:55.913189 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:27:55.913970 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:55.904145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46295 (local address 127.19.65.129:53137)
0903 14:27:55.904490 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:55.904518 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:55.905107 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:55.905566 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:55.905573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:55.905946 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:55.906803 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:55.906815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.907507 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:55.907510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:55.907871 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:55.907878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:55.908043 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:55.908552 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:55.908574 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:55.910153 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:55.911724 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:55.911729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:55.911738 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:55.912035 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:55.912240 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:55.912246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:55.912250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:55.912347 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:55.912634 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:55.912640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:55.912879 (+ 239us) client_negotiation.cc:769] Sending connection context
0903 14:27:55.913092 (+ 213us) client_negotiation.cc:240] Negotiation successful
0903 14:27:55.913318 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20250903 14:27:55.914906 30816 heartbeater.cc:344] Connected to a master server at 127.19.65.190:46295
I20250903 14:27:55.915092 30816 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:55.915535 30816 heartbeater.cc:507] Master 127.19.65.190:46295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
I20250903 14:27:55.916749 30623 ts_manager.cc:194] Registered new tserver with Master: b26d755c2aee4762a3b27bc5c3d6fc65 (127.19.65.129:43135)
I20250903 14:27:55.917779 30623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:53137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:55.942202 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46295
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:56.029844 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:56.029991 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:56.030068 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:56.032931 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:56.032985 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:56.033085 30823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:27:56.036643 30823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:56.037372 30823 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:56.038053 30823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:56.043109 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:56.043133 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:56.043216 30823 server_base.cc:1047] running on GCE node
W20250903 14:27:56.043102 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:56.043527 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:56.043936 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:56.045071 30823 hybrid_clock.cc:648] HybridClock initialized: now 1756909676045058 us; error 21 us; skew 500 ppm
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909676, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:56.047091 30823 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:27:56.047847 30823 webserver.cc:480] Webserver started at http://127.19.65.130:44963/ using document root <none> and password file <none>
I20250903 14:27:56.048317 30823 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:56.048370 30823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:56.048537 30823 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:56.049969 30823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "fff8fbbf361f4c4797e7163f5a78464b"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "392696d30ba733501e35c127c15e2073"
server_key_iv: "ba56e81c45d4645be1ea5de9dd725939"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.050349 30823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "fff8fbbf361f4c4797e7163f5a78464b"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "392696d30ba733501e35c127c15e2073"
server_key_iv: "ba56e81c45d4645be1ea5de9dd725939"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.052910 30823 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:27:56.054693 30838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.055444 30823 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 14:27:56.055526 30823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "fff8fbbf361f4c4797e7163f5a78464b"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "392696d30ba733501e35c127c15e2073"
server_key_iv: "ba56e81c45d4645be1ea5de9dd725939"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.055595 30823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:56.074366 30823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:56.076321 30823 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:56.076457 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:56.076840 30823 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:56.077456 30823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:56.077535 30823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.077575 30823 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:56.077603 30823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.084770 30823 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:39657
I20250903 14:27:56.084789 30951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:39657 every 8 connection(s)
I20250903 14:27:56.085459 30823 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:27:56.085942 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30823
I20250903 14:27:56.086051 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:27:56.086282 19718 external_mini_cluster.cc:1468] Setting key 130cbcf9218d197a341feb0deb740a59
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909676, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20250903 14:27:56.095922 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.086885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:32885 (local address 127.19.65.190:46295)
0903 14:27:56.087019 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.087023 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.087637 (+ 614us) server_negotiation.cc:408] Connection header received
0903 14:27:56.088408 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.088414 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.088485 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.088575 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:56.089950 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.090377 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.090972 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.091163 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.092976 (+ 1813us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:56.092990 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:56.092992 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:56.093013 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:56.094340 (+ 1327us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.094719 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.094724 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.094725 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.094762 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.095003 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.095006 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.095007 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.095146 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:56.095241 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.095682 (+ 441us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.095785 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
I20250903 14:27:56.096393 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.087090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46295 (local address 127.19.65.130:32885)
0903 14:27:56.087523 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.087563 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.088232 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.088697 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.088703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.089038 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:56.089825 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.089837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.090483 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.090485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.090873 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.090879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.091034 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.091595 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:56.091614 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:56.092849 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:56.094428 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:56.094432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:56.094447 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:56.094632 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:56.094859 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:56.094861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:56.094863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:56.094928 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:56.095242 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:56.095248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:56.095438 (+ 190us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.095637 (+ 199us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.095837 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20250903 14:27:56.097329 30952 heartbeater.cc:344] Connected to a master server at 127.19.65.190:46295
I20250903 14:27:56.097566 30952 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:56.098075 30952 heartbeater.cc:507] Master 127.19.65.190:46295 requested a full tablet report, sending...
I20250903 14:27:56.098968 30623 ts_manager.cc:194] Registered new tserver with Master: fff8fbbf361f4c4797e7163f5a78464b (127.19.65.130:39657)
I20250903 14:27:56.099452 30623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:32885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:27:56.123520 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46295
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:27:56.211226 30959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:27:56.211381 30959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:27:56.211464 30959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:27:56.214043 30959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:27:56.214090 30959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:27:56.214193 30959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20250903 14:27:56.217448 30959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:27:56.218331 30959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:27:56.219069 30959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:27:56.223986 30965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:27:56.224050 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:56.224169 30959 server_base.cc:1047] running on GCE node
W20250903 14:27:56.223986 30964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:27:56.224526 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:27:56.224908 30959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:27:56.226033 30959 hybrid_clock.cc:648] HybridClock initialized: now 1756909676226021 us; error 25 us; skew 500 ppm
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909676, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:27:56.228194 30959 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20250903 14:27:56.229068 30959 webserver.cc:480] Webserver started at http://127.19.65.131:36513/ using document root <none> and password file <none>
I20250903 14:27:56.229549 30959 fs_manager.cc:362] Metadata directory not provided
I20250903 14:27:56.229595 30959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:27:56.229744 30959 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:27:56.231153 30959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/instance:
uuid: "32b180a1d3f44e679369022f1c001389"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "df84e211505d9fb4c535453b8104f3c6"
server_key_iv: "7585c83878f8587efd2223b289139d63"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.231556 30959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "32b180a1d3f44e679369022f1c001389"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "df84e211505d9fb4c535453b8104f3c6"
server_key_iv: "7585c83878f8587efd2223b289139d63"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.234290 30959 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250903 14:27:56.236032 30974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.236811 30959 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 14:27:56.236886 30959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
uuid: "32b180a1d3f44e679369022f1c001389"
format_stamp: "Formatted at 2025-09-03 14:27:56 on dist-test-slave-4wpb"
server_key: "df84e211505d9fb4c535453b8104f3c6"
server_key_iv: "7585c83878f8587efd2223b289139d63"
server_key_version: "encryptionkey@0"
I20250903 14:27:56.236959 30959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:27:56.249739 30959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:27:56.251947 30959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:27:56.252058 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:27:56.252439 30959 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:27:56.253055 30959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:27:56.253094 30959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.253131 30959 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:27:56.253159 30959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:27:56.260733 30959 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:44391
I20250903 14:27:56.260756 31087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.131:44391 every 8 connection(s)
I20250903 14:27:56.261412 30959 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/data/info.pb
I20250903 14:27:56.267372 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 30959
I20250903 14:27:56.267458 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/instance
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909676, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:27:56.267652 19718 external_mini_cluster.cc:1468] Setting key f5aec83b7a77b59eef1f6f11ab2ed9ec
I20250903 14:27:56.270915 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.262665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:47531 (local address 127.19.65.190:46295)
0903 14:27:56.262778 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.262781 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.263258 (+ 477us) server_negotiation.cc:408] Connection header received
0903 14:27:56.263904 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.263908 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.263959 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.264028 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:56.265270 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.265735 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.266251 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.266410 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.268139 (+ 1729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:56.268152 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:56.268155 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:56.268181 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:56.269639 (+ 1458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.270008 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.270011 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.270012 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.270046 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.270207 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.270209 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.270210 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.270319 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:56.270385 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.270712 (+ 327us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.270810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 14:27:56.271414 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.262835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46295 (local address 127.19.65.131:47531)
0903 14:27:56.263151 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.263179 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.263750 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.264153 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.264160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.264442 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:27:56.265141 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.265152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.265838 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.265841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.266156 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.266162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.266309 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.266735 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:27:56.266753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:27:56.267989 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:27:56.269713 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:56.269717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:56.269741 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:56.269940 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:56.270095 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:27:56.270097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:27:56.270098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:27:56.270154 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:27:56.270379 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:27:56.270382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:27:56.270542 (+ 160us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.270672 (+ 130us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.270852 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20250903 14:27:56.272220 31088 heartbeater.cc:344] Connected to a master server at 127.19.65.190:46295
I20250903 14:27:56.272383 31088 heartbeater.cc:461] Registering TS with master...
I20250903 14:27:56.272738 31088 heartbeater.cc:507] Master 127.19.65.190:46295 requested a full tablet report, sending...
I20250903 14:27:56.273468 30623 ts_manager.cc:194] Registered new tserver with Master: 32b180a1d3f44e679369022f1c001389 (127.19.65.131:44391)
I20250903 14:27:56.273937 30623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:47531
I20250903 14:27:56.281116 19718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:27:56.288380 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.282689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.19.65.190:46295)
0903 14:27:56.282816 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.282820 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.282966 (+ 146us) server_negotiation.cc:408] Connection header received
0903 14:27:56.283073 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.283076 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.283114 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.283182 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:27:56.283891 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.284294 (+ 403us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.284858 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.285037 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.285913 (+ 876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:27:56.285929 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:27:56.285931 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:27:56.285950 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:27:56.287065 (+ 1115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.287419 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.287423 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.287424 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.287469 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:27:56.287666 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:27:56.287670 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:27:56.287671 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:27:56.287802 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:27:56.287943 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.288138 (+ 195us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.288242 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 14:27:56.290561 30623 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:27:56.292697 30623 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:27:56.303042 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.299097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:44391 (local address 127.0.0.1:47808)
0903 14:27:56.299552 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.299586 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.299711 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.300515 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.300520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.300542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:56.300779 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.300787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.302014 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.302016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.302744 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.302752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.302832 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.302845 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.302886 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.302930 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":58,"threads_started":1}
I20250903 14:27:56.303383 31101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.299245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39657 (local address 127.0.0.1:37312)
0903 14:27:56.299789 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.299803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.299900 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.300601 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.300604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.300618 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:56.300847 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.300856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.301803 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.301806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.302521 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.302529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.302628 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.302666 (+ 38us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.302731 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.302810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":76,"threads_started":1}
I20250903 14:27:56.303471 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.299746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.19.65.130:39657)
0903 14:27:56.300078 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.300085 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.300109 (+ 24us) server_negotiation.cc:408] Connection header received
0903 14:27:56.300180 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.300184 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.300339 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.300464 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:56.300969 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.301713 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.302658 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.303111 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.303213 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.303274 (+ 61us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.303349 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":141,"threads_started":1}
I20250903 14:27:56.303711 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.299698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43135 (local address 127.0.0.1:58438)
0903 14:27:56.299983 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.299995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.300065 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.300901 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.300906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.300927 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:56.301135 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.301141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.302253 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.302257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.302932 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.302943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.303041 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.303057 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.303100 (+ 43us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.303598 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20250903 14:27:56.304306 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.299191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.19.65.131:44391)
0903 14:27:56.299505 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.299511 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.300119 (+ 608us) server_negotiation.cc:408] Connection header received
0903 14:27:56.300187 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.300192 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.300348 (+ 156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.300899 (+ 551us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:56.301001 (+ 102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.301919 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.303582 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.304008 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.304084 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.304137 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.304215 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":91,"threads_started":1}
I20250903 14:27:56.304936 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.300168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.19.65.129:43135)
0903 14:27:56.300543 (+ 375us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.300549 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.300567 (+ 18us) server_negotiation.cc:408] Connection header received
0903 14:27:56.300638 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.300642 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.300767 (+ 125us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.300886 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:56.301245 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.302157 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.303265 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.304087 (+ 822us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.304152 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.304208 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.304277 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":142,"threads_started":1}
I20250903 14:27:56.305719 30886 tablet_service.cc:1468] Processing CreateTablet for tablet f28837c243424c01a365af60c241f78d (DEFAULT_TABLE table=test-table [id=dfc2bf7cc99d44d4bbc0b40d4764de2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:56.306087 31022 tablet_service.cc:1468] Processing CreateTablet for tablet f28837c243424c01a365af60c241f78d (DEFAULT_TABLE table=test-table [id=dfc2bf7cc99d44d4bbc0b40d4764de2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:56.306514 30886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28837c243424c01a365af60c241f78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:56.306874 31022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28837c243424c01a365af60c241f78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:56.308070 30750 tablet_service.cc:1468] Processing CreateTablet for tablet f28837c243424c01a365af60c241f78d (DEFAULT_TABLE table=test-table [id=dfc2bf7cc99d44d4bbc0b40d4764de2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:27:56.308862 30750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28837c243424c01a365af60c241f78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:27:56.311336 31105 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Bootstrap starting.
I20250903 14:27:56.312311 31106 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Bootstrap starting.
I20250903 14:27:56.313556 31105 tablet_bootstrap.cc:654] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:56.313793 31107 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Bootstrap starting.
I20250903 14:27:56.314208 31106 tablet_bootstrap.cc:654] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:56.314563 31105 log.cc:826] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:56.315032 31106 log.cc:826] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:56.315640 31107 tablet_bootstrap.cc:654] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Neither blocks nor log segments found. Creating new log.
I20250903 14:27:56.316309 31105 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: No bootstrap required, opened a new log
I20250903 14:27:56.316322 31107 log.cc:826] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Log is configured to *not* fsync() on all Append() calls
I20250903 14:27:56.316375 31106 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: No bootstrap required, opened a new log
I20250903 14:27:56.316486 31105 ts_tablet_manager.cc:1397] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250903 14:27:56.316565 31106 ts_tablet_manager.cc:1397] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:27:56.317809 31107 tablet_bootstrap.cc:492] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: No bootstrap required, opened a new log
I20250903 14:27:56.317986 31107 ts_tablet_manager.cc:1397] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250903 14:27:56.318997 31106 raft_consensus.cc:357] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.319218 31106 raft_consensus.cc:383] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:56.319274 31106 raft_consensus.cc:738] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b180a1d3f44e679369022f1c001389, State: Initialized, Role: FOLLOWER
I20250903 14:27:56.319620 31106 consensus_queue.cc:260] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.319718 31105 raft_consensus.cc:357] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.319880 31105 raft_consensus.cc:383] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:56.319940 31105 raft_consensus.cc:738] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff8fbbf361f4c4797e7163f5a78464b, State: Initialized, Role: FOLLOWER
I20250903 14:27:56.320281 31105 consensus_queue.cc:260] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.320449 31106 ts_tablet_manager.cc:1428] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 14:27:56.320744 31088 heartbeater.cc:499] Master 127.19.65.190:46295 was elected leader, sending a full tablet report...
I20250903 14:27:56.321089 31105 ts_tablet_manager.cc:1428] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20250903 14:27:56.321230 31107 raft_consensus.cc:357] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.321385 31107 raft_consensus.cc:383] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:27:56.321429 31107 raft_consensus.cc:738] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26d755c2aee4762a3b27bc5c3d6fc65, State: Initialized, Role: FOLLOWER
I20250903 14:27:56.321525 30952 heartbeater.cc:499] Master 127.19.65.190:46295 was elected leader, sending a full tablet report...
I20250903 14:27:56.321815 31107 consensus_queue.cc:260] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.322395 30816 heartbeater.cc:499] Master 127.19.65.190:46295 was elected leader, sending a full tablet report...
I20250903 14:27:56.322597 31107 ts_tablet_manager.cc:1428] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20250903 14:27:56.336078 30953 tablet.cc:2378] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 14:27:56.403617 30817 tablet.cc:2378] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:56.502805 31112 raft_consensus.cc:491] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:56.503005 31112 raft_consensus.cc:513] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.504062 31112 leader_election.cc:290] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32b180a1d3f44e679369022f1c001389 (127.19.65.131:44391), b26d755c2aee4762a3b27bc5c3d6fc65 (127.19.65.129:43135)
I20250903 14:27:56.507246 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.504277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:44391 (local address 127.19.65.130:39009)
0903 14:27:56.504445 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.504462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.504561 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.504833 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.504837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.504853 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:56.505092 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.505100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.506017 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.506020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.506836 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.506845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.506954 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.506974 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.507025 (+ 51us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.507095 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20250903 14:27:56.507367 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.504277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43135 (local address 127.19.65.130:60215)
0903 14:27:56.504741 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.504752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 14:27:56.504848 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:27:56.505081 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:27:56.505084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:27:56.505100 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 14:27:56.505328 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.505333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.506230 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.506234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:27:56.506999 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:27:56.507010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.507104 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.507115 (+ 11us) client_negotiation.cc:769] Sending connection context
0903 14:27:56.507160 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 14:27:56.507203 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":126,"thread_start_us":161,"threads_started":1}
I20250903 14:27:56.507663 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.504394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:39009 (local address 127.19.65.131:44391)
0903 14:27:56.504621 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.504626 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.504642 (+ 16us) server_negotiation.cc:408] Connection header received
0903 14:27:56.504695 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.504698 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.504740 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.504811 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:56.505223 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.505910 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.506952 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.507412 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.507449 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.507499 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.507550 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20250903 14:27:56.507819 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.504438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60215 (local address 127.19.65.129:43135)
0903 14:27:56.504568 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.504573 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.504834 (+ 261us) server_negotiation.cc:408] Connection header received
0903 14:27:56.504947 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.504949 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.504988 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.505046 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:27:56.505437 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.506117 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.507122 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.507576 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.507609 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.507654 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.507714 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:27:56.508219 31042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28837c243424c01a365af60c241f78d" candidate_uuid: "fff8fbbf361f4c4797e7163f5a78464b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b180a1d3f44e679369022f1c001389" is_pre_election: true
I20250903 14:27:56.508312 30770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28837c243424c01a365af60c241f78d" candidate_uuid: "fff8fbbf361f4c4797e7163f5a78464b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" is_pre_election: true
I20250903 14:27:56.508486 31042 raft_consensus.cc:2466] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fff8fbbf361f4c4797e7163f5a78464b in term 0.
I20250903 14:27:56.508549 30770 raft_consensus.cc:2466] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fff8fbbf361f4c4797e7163f5a78464b in term 0.
I20250903 14:27:56.508893 30839 leader_election.cc:304] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b180a1d3f44e679369022f1c001389, fff8fbbf361f4c4797e7163f5a78464b; no voters:
I20250903 14:27:56.509109 31112 raft_consensus.cc:2802] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:27:56.509176 31112 raft_consensus.cc:491] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:27:56.509230 31112 raft_consensus.cc:3058] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:56.510121 31112 raft_consensus.cc:513] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.510419 31112 leader_election.cc:290] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [CANDIDATE]: Term 1 election: Requested vote from peers 32b180a1d3f44e679369022f1c001389 (127.19.65.131:44391), b26d755c2aee4762a3b27bc5c3d6fc65 (127.19.65.129:43135)
I20250903 14:27:56.510708 31042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28837c243424c01a365af60c241f78d" candidate_uuid: "fff8fbbf361f4c4797e7163f5a78464b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b180a1d3f44e679369022f1c001389"
I20250903 14:27:56.510715 30770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28837c243424c01a365af60c241f78d" candidate_uuid: "fff8fbbf361f4c4797e7163f5a78464b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65"
I20250903 14:27:56.510797 30770 raft_consensus.cc:3058] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:56.510802 31042 raft_consensus.cc:3058] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:27:56.511644 31042 raft_consensus.cc:2466] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fff8fbbf361f4c4797e7163f5a78464b in term 1.
I20250903 14:27:56.511708 30770 raft_consensus.cc:2466] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fff8fbbf361f4c4797e7163f5a78464b in term 1.
I20250903 14:27:56.511920 30839 leader_election.cc:304] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b180a1d3f44e679369022f1c001389, fff8fbbf361f4c4797e7163f5a78464b; no voters:
I20250903 14:27:56.512053 31112 raft_consensus.cc:2802] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 1 FOLLOWER]: Leader election won for term 1
W20250903 14:27:56.512085 31089 tablet.cc:2378] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:27:56.512272 31112 raft_consensus.cc:695] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 1 LEADER]: Becoming Leader. State: Replica: fff8fbbf361f4c4797e7163f5a78464b, State: Running, Role: LEADER
I20250903 14:27:56.512542 31112 consensus_queue.cc:237] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } }
I20250903 14:27:56.514915 30623 catalog_manager.cc:5582] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b reported cstate change: term changed from 0 to 1, leader changed from <none> to fff8fbbf361f4c4797e7163f5a78464b (127.19.65.130). New cstate: current_term: 1 leader_uuid: "fff8fbbf361f4c4797e7163f5a78464b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fff8fbbf361f4c4797e7163f5a78464b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 } health_report { overall_health: UNKNOWN } } }
I20250903 14:27:56.532857 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.530372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.19.65.130:39657)
0903 14:27:56.530469 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0903 14:27:56.530472 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 14:27:56.530483 (+ 11us) server_negotiation.cc:408] Connection header received
0903 14:27:56.530623 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:27:56.530625 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:27:56.530667 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:27:56.530741 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 14:27:56.531007 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.531419 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:27:56.532005 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:27:56.532136 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:27:56.532208 (+ 72us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 14:27:56.532585 (+ 377us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 14:27:56.532673 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 14:27:56.532740 (+ 67us) server_negotiation.cc:300] Negotiation successful
0903 14:27:56.532778 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250903 14:27:56.537325 31042 raft_consensus.cc:1273] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 1 FOLLOWER]: Refusing update from remote peer fff8fbbf361f4c4797e7163f5a78464b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:56.537345 30770 raft_consensus.cc:1273] T f28837c243424c01a365af60c241f78d P b26d755c2aee4762a3b27bc5c3d6fc65 [term 1 FOLLOWER]: Refusing update from remote peer fff8fbbf361f4c4797e7163f5a78464b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:27:56.537844 31112 consensus_queue.cc:1035] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b26d755c2aee4762a3b27bc5c3d6fc65" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:56.537989 31115 consensus_queue.cc:1035] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b180a1d3f44e679369022f1c001389" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:27:56.545269 31120 mvcc.cc:204] Tried to move back new op lower bound from 7196302035091664896 to 7196302034997284864. Current Snapshot: MvccSnapshot[applied={T|T < 7196302035091664896}]
I20250903 14:27:56.551824 30623 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:56.551954 30623 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 14:27:56.553972 30623 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b2d8004e32374bf591372cc50a8dc8ee: Sending DeleteTablet for 3 replicas of tablet f28837c243424c01a365af60c241f78d
I20250903 14:27:56.554558 31022 tablet_service.cc:1515] Processing DeleteTablet for tablet f28837c243424c01a365af60c241f78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:56 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:47808
I20250903 14:27:56.554697 30886 tablet_service.cc:1515] Processing DeleteTablet for tablet f28837c243424c01a365af60c241f78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:56 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:37312
I20250903 14:27:56.554795 30750 tablet_service.cc:1515] Processing DeleteTablet for tablet f28837c243424c01a365af60c241f78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:27:56 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:58438
I20250903 14:27:56.554971 31128 tablet_replica.cc:331] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: stopping tablet replica
I20250903 14:27:56.555248 31128 raft_consensus.cc:2241] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:27:56.555441 31129 tablet_replica.cc:331] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: stopping tablet replica
I20250903 14:27:56.555487 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30687
I20250903 14:27:56.555511 31128 raft_consensus.cc:2270] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:56.557340 31128 ts_tablet_manager.cc:1905] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:56.558136 31129 raft_consensus.cc:2241] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:27:56.558450 31129 raft_consensus.cc:2270] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:27:56.559716 31129 ts_tablet_manager.cc:1905] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:27:56.559826 31128 ts_tablet_manager.cc:1918] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:56.559970 31128 log.cc:1199] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-2/wal/wals/f28837c243424c01a365af60c241f78d
I20250903 14:27:56.560362 31128 ts_tablet_manager.cc:1939] T f28837c243424c01a365af60c241f78d P 32b180a1d3f44e679369022f1c001389: Deleting consensus metadata
I20250903 14:27:56.561376 30607 catalog_manager.cc:4928] TS 32b180a1d3f44e679369022f1c001389 (127.19.65.131:44391): tablet f28837c243424c01a365af60c241f78d (table test-table [id=dfc2bf7cc99d44d4bbc0b40d4764de2b]) successfully deleted
I20250903 14:27:56.562749 31129 ts_tablet_manager.cc:1918] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:27:56.562834 31129 log.cc:1199] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Deleting WAL directory at /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756909585842267-19718-0/minicluster-data/ts-1/wal/wals/f28837c243424c01a365af60c241f78d
I20250903 14:27:56.563165 31129 ts_tablet_manager.cc:1939] T f28837c243424c01a365af60c241f78d P fff8fbbf361f4c4797e7163f5a78464b: Deleting consensus metadata
I20250903 14:27:56.564137 30609 catalog_manager.cc:4928] TS fff8fbbf361f4c4797e7163f5a78464b (127.19.65.130:39657): tablet f28837c243424c01a365af60c241f78d (table test-table [id=dfc2bf7cc99d44d4bbc0b40d4764de2b]) successfully deleted
W20250903 14:27:56.564337 30609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:43135 (error 108)
I20250903 14:27:56.564559 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30823
I20250903 14:27:56.566262 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:27:56.565957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43135 (local address 127.0.0.1:58452)
0903 14:27:56.566094 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0903 14:27:56.566157 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20250903 14:27:56.566404 30609 catalog_manager.cc:4671] TS b26d755c2aee4762a3b27bc5c3d6fc65 (127.19.65.129:43135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f28837c243424c01a365af60c241f78d: Network error: Client connection negotiation failed: client connection to 127.19.65.129:43135: connect: Connection refused (error 111)
I20250903 14:27:56.570284 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30959
I20250903 14:27:56.575373 19718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1dJ511/build/debug/bin/kudu with pid 30591
2025-09-03T14:27:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2927 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[31134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[31134](info): set up 2 sockets
Sep 03 14:27:56 dist-test-slave-4wpb krb5kdc[31134](info): commencing operation
krb5kdc: starting...
W20250903 14:27:58.329909 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.733s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/test-user.keytab.
Sep 03 14:27:58 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T14:27:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:27:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
I20250903 14:27:58.453141 19718 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1dJ511/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/postgres -l logfile start
2025-09-03 14:28:00.525 UTC [31165] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-03 14:28:00.526 UTC [31165] LOG: listening on IPv4 address "127.19.65.148", port 47505
2025-09-03 14:28:00.528 UTC [31165] LOG: listening on Unix socket "/tmp/.s.PGSQL.47505"
2025-09-03 14:28:00.533 UTC [31170] LOG: database system was shut down at 2025-09-03 14:27:59 UTC
2025-09-03 14:28:00.537 UTC [31165] LOG: database system is ready to accept connections
I20250903 14:28:02.233377 19718 mini_postgres.cc:96] Postgres bound to 47505
2025-09-03 14:28:02.238 UTC [31176] FATAL: database "slave" does not exist
127.19.65.148:47505 - accepting connections
I20250903 14:28:02.239342 19718 mini_ranger.cc:161] Starting Ranger...
I20250903 14:28:02.255038 19718 mini_ranger.cc:84] Created miniranger Postgres user
I20250903 14:28:02.314751 19718 mini_ranger.cc:87] Created ranger Postgres database
I20250903 14:28:02.314834 19718 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin
2025-09-03 14:28:02,575 [I] DB FLAVOR :POSTGRES
2025-09-03 14:28:02,576 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 14:28:02,576 [I] Checking connection..
2025-09-03 14:28:02,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-03 14:28:02,856 [I] Checking connection passed.
2025-09-03 14:28:02,856 [I] --------- Verifying version history table ---------
2025-09-03 14:28:02,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 14:28:03,143 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 14:28:03,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 14:28:03,406 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 14:28:03,406 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-03 14:28:03,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-03 14:28:03.667 UTC [31243] WARNING: there is no transaction in progress
2025-09-03 14:28:03,673 [I] create_dbversion_catalog.sql file imported successfully
2025-09-03 14:28:03,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 14:28:03,960 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-03 14:28:03,960 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 14:28:03,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-03 14:28:04,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-03 14:28:04,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4wpb.c.gcp-upstream.internal','N') ;"
2025-09-03 14:28:04,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 14:28:05,003 [I] Table x_portal_user does not exist in database ranger
2025-09-03 14:28:05,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 14:28:05,291 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-03 14:28:05,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 14:28:05,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 14:28:05,888 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-03 14:28:05,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-03 14:28:06.608 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.617 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.626 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.635 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.645 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.654 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.692 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.696 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.706 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.715 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.723 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.731 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.737 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.741 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.744 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:06.747 UTC [31360] WARNING: there is no transaction in progress
2025-09-03 14:28:07,088 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-03 14:28:07,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-03 14:28:07,364 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-03 14:28:07,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 14:28:07,626 [I] Table x_portal_user already exists in database 'ranger'
2025-09-03 14:28:07,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 14:28:07,886 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-03 14:28:07,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 14:28:08,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 14:28:08,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4wpb.c.gcp-upstream.internal';"
2025-09-03 14:28:08,719 [I] CORE_DB_SCHEMA import status has been updated
2025-09-03 14:28:08,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-03 14:28:08,998 [I] DB_PATCHES have already been applied
I20250903 14:28:09.003067 19718 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1dJ511/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250903 14:28:09.003120 19718 mini_ranger.cc:193] Using host: 127.19.65.148
I20250903 14:28:09.005676 19718 mini_ranger.cc:227] Ranger admin URL: http://127.19.65.148:45185
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin/ews/webapp
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 03, 2025 2:28:09 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-admin/ews/webapp].
Sep 03, 2025 2:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 03, 2025 2:28:09 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-45185"]
Sep 03, 2025 2:28:09 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 2:28:09 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 2:28:10 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 03, 2025 2:28:10 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 03, 2025 2:28:10 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 03, 2025 2:28:10 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 03, 2025 2:28:10 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03, 2025 2:28:10 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Sep 03, 2025 2:28:10 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20250903 14:28:10.991212 19718 mini_ranger.cc:160] Time spent starting Ranger: real 8.752s user 0.002s sys 0.006s
Sep 03, 2025 2:28:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 03, 2025 2:28:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Sep 03, 2025 2:28:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 03, 2025 2:28:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 03, 2025 2:28:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 03, 2025 2:28:28 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-45185"]
I20250903 14:28:29.069900 19718 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.65.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 03 14:28:29 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250903 14:28:29.138984 19718 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250903 14:28:29.147421 19718 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250903 14:28:29.244802 19718 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250903 14:28:29.244899 19718 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms
I20250903 14:28:29.244938 19718 mini_ranger_kms.cc:227] Using postgres at 127.19.65.148:47505
2025-09-03 14:28:29,412 [I] DB FLAVOR :POSTGRES
2025-09-03 14:28:29,413 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 14:28:29,413 [I] Checking connection
2025-09-03 14:28:29,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-03 14:28:29,680 [I] connection success
2025-09-03 14:28:29,680 [I] --------- Verifying Ranger DB tables ---------
2025-09-03 14:28:29,680 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-03 14:28:29,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-03 14:28:29,950 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-03 14:28:29,951 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 14:28:29,951 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-03 14:28:29,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-03 14:28:30,224 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250903 14:28:30.638262 19718 mini_ranger_kms.cc:310] Created kms service
I20250903 14:28:30.813817 19718 mini_ranger_kms.cc:326] Created kudu user
I20250903 14:28:30.847724 19718 mini_ranger_kms.cc:343] Created rangerkms user
Sep 03 14:28:30 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20250903 14:28:31.147954 19718 mini_ranger_kms.cc:384] Added ranger policy
I20250903 14:28:31.148142 19718 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms:/tmp/dist-test-task1dJ511/build/debug/bin/postgresql.jar:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1dJ511/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250903 14:28:31.148168 19718 mini_ranger_kms.cc:240] Using host: 127.19.65.148
I20250903 14:28:31.150911 19718 mini_ranger_kms.cc:276] Ranger KMS PID: 31552
I20250903 14:28:31.150998 19718 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.65.148:45265
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-kms/ews/webapp].
Sep 03, 2025 2:28:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 03, 2025 2:28:32 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-45265"]
Sep 03, 2025 2:28:32 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 2:28:32 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 2:28:33 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20250903 14:28:33.193440 19718 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.054s user 0.002s sys 0.151s
I20250903 14:28:33.193603 19718 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250903 14:28:33.193689 19718 mini_ranger_kms.cc:401] 127.19.65.148:45265/kms/v1/keys
Sep 03, 2025 2:28:33 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03 14:28:33 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1756909713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 03 14:28:33 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Sep 03 14:28:34 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Sep 03 14:28:36 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:28:36.062064 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32985
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:32985
--ranger_config_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:45265/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:28:36.153877 31605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:28:36.154023 31605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:28:36.154063 31605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:28:36.157379 31605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:28:36.157426 31605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:28:36.157445 31605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:28:36.157460 31605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:28:36.157503 31605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:28:36.161062 31605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.65.148:45265/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:32985
--ranger_config_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:28:36.162078 31605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:28:36.162855 31605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:28:36.167579 31613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:28:36.167619 31610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:28:36.167814 31605 server_base.cc:1047] running on GCE node
W20250903 14:28:36.167816 31611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:28:36.168339 31605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:28:36.169236 31605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:28:36.170390 31605 hybrid_clock.cc:648] HybridClock initialized: now 1756909716170378 us; error 25 us; skew 500 ppm
Sep 03 14:28:36 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909716, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:28:36.172978 31605 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20250903 14:28:36.173859 31605 webserver.cc:480] Webserver started at http://127.19.65.190:33605/ using document root <none> and password file <none>
I20250903 14:28:36.174268 31605 fs_manager.cc:362] Metadata directory not provided
I20250903 14:28:36.174314 31605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:28:36.174464 31605 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 14:28:36 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909716, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20250903 14:28:36.226486 31605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data/instance:
uuid: "c6b371d488de44689074a2a8d9d95b71"
format_stamp: "Formatted at 2025-09-03 14:28:36 on dist-test-slave-4wpb"
server_key: "b186aeb30a6499f0ff0512b96629b38b"
server_key_iv: "cc467aadd43d5945f29da835cab350c1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:36.226913 31605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal/instance:
uuid: "c6b371d488de44689074a2a8d9d95b71"
format_stamp: "Formatted at 2025-09-03 14:28:36 on dist-test-slave-4wpb"
server_key: "b186aeb30a6499f0ff0512b96629b38b"
server_key_iv: "cc467aadd43d5945f29da835cab350c1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:36.229977 31605 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:28:36.244186 31620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:36.245074 31605 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:28:36.245163 31605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal
uuid: "c6b371d488de44689074a2a8d9d95b71"
format_stamp: "Formatted at 2025-09-03 14:28:36 on dist-test-slave-4wpb"
server_key: "b186aeb30a6499f0ff0512b96629b38b"
server_key_iv: "cc467aadd43d5945f29da835cab350c1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:36.245244 31605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:28:36.272519 31605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:28:36.285542 31605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:28:36.285693 31605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:28:36.293560 31605 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:32985
I20250903 14:28:36.293565 31672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.190:32985 every 8 connection(s)
I20250903 14:28:36.294448 31605 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/data/info.pb
I20250903 14:28:36.296721 31673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:28:36.297014 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 31605
I20250903 14:28:36.297137 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/wal/instance
I20250903 14:28:36.297370 19718 external_mini_cluster.cc:1468] Setting key 9bac8499204eb3dad52f38934c0399a1
I20250903 14:28:36.302201 31673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71: Bootstrap starting.
I20250903 14:28:36.304505 31673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71: Neither blocks nor log segments found. Creating new log.
Sep 03 14:28:36 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:28:36.305766 31673 log.cc:826] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71: Log is configured to *not* fsync() on all Append() calls
I20250903 14:28:36.307672 31673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71: No bootstrap required, opened a new log
I20250903 14:28:36.309415 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:36.298521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.19.65.190:32985)
0903 14:28:36.298905 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0903 14:28:36.298915 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 14:28:36.298951 (+ 36us) server_negotiation.cc:408] Connection header received
0903 14:28:36.299504 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:28:36.299525 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:28:36.299838 (+ 313us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:28:36.300124 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:28:36.300994 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:28:36.301969 (+ 975us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:28:36.302870 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:28:36.303138 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:28:36.305543 (+ 2405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:28:36.305561 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:28:36.305575 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:28:36.305607 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:28:36.307508 (+ 1901us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:28:36.307868 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:28:36.307872 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:28:36.307876 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:28:36.307927 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:28:36.308112 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:28:36.308115 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:28:36.308116 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:28:36.308366 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:28:36.308521 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0903 14:28:36.308750 (+ 229us) server_negotiation.cc:300] Negotiation successful
0903 14:28:36.308941 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20250903 14:28:36.310227 31673 raft_consensus.cc:357] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } }
I20250903 14:28:36.310436 31673 raft_consensus.cc:383] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:28:36.310527 31673 raft_consensus.cc:738] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b371d488de44689074a2a8d9d95b71, State: Initialized, Role: FOLLOWER
I20250903 14:28:36.310968 31673 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } }
I20250903 14:28:36.311139 31673 raft_consensus.cc:397] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 14:28:36.311203 31673 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 14:28:36.311316 31673 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:28:36.312202 31673 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } }
I20250903 14:28:36.312552 31673 leader_election.cc:304] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b371d488de44689074a2a8d9d95b71; no voters:
I20250903 14:28:36.312866 31673 leader_election.cc:290] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 14:28:36.312938 31678 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:28:36.313135 31678 raft_consensus.cc:695] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [term 1 LEADER]: Becoming Leader. State: Replica: c6b371d488de44689074a2a8d9d95b71, State: Running, Role: LEADER
I20250903 14:28:36.313535 31678 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } }
I20250903 14:28:36.313933 31673 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:28:36.314380 31678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b371d488de44689074a2a8d9d95b71. Latest consensus state: current_term: 1 leader_uuid: "c6b371d488de44689074a2a8d9d95b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } } }
I20250903 14:28:36.314584 31678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [sys.catalog]: This master's current role is: LEADER
I20250903 14:28:36.314759 31679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b371d488de44689074a2a8d9d95b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b371d488de44689074a2a8d9d95b71" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32985 } } }
I20250903 14:28:36.314836 31679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71 [sys.catalog]: This master's current role is: LEADER
I20250903 14:28:36.315021 31685 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:28:36.318125 31685 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:28:36.318548 31673 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250903 14:28:36.327840 31685 catalog_manager.cc:1349] Generated new cluster ID: 8f1aeba94fdd499db8d07024ea1ae1d8
I20250903 14:28:36.327908 31685 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:28:36.354851 31685 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:28:36.355688 31685 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:28:36.369413 31685 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c6b371d488de44689074a2a8d9d95b71: Generated new TSK 0
I20250903 14:28:36.370020 31685 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
I20250903 14:28:36.986078 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:45265/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32985
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 14:28:37.093683 31719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:28:37.093832 31719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:28:37.093876 31719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:28:37.096572 31719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:28:37.096623 31719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:28:37.096711 31719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20250903 14:28:37.099999 31719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.65.148:45265/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:32985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:28:37.100776 31719 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:28:37.101439 31719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:28:37.106587 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:28:37.106587 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:28:37.106951 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:28:37.107023 31719 server_base.cc:1047] running on GCE node
I20250903 14:28:37.107295 31719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:28:37.107725 31719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:28:37.108914 31719 hybrid_clock.cc:648] HybridClock initialized: now 1756909717108901 us; error 30 us; skew 500 ppm
Sep 03 14:28:37 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:28:37.111101 31719 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20250903 14:28:37.111897 31719 webserver.cc:480] Webserver started at http://127.19.65.129:35401/ using document root <none> and password file <none>
I20250903 14:28:37.112371 31719 fs_manager.cc:362] Metadata directory not provided
I20250903 14:28:37.112422 31719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:28:37.112596 31719 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 14:28:37 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20250903 14:28:37.132457 31719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data/instance:
uuid: "3eee1ec72f6d487988d73e2e7c3b9023"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "ab651f82cceba880edce90e14d145a67"
server_key_iv: "3e5a9424f99cc229ca1c0b406a201340"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.132908 31719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "3eee1ec72f6d487988d73e2e7c3b9023"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "ab651f82cceba880edce90e14d145a67"
server_key_iv: "3e5a9424f99cc229ca1c0b406a201340"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.138566 31719 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 14:28:37.165781 31736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.166779 31719 fs_manager.cc:730] Time spent opening block manager: real 0.017s user 0.001s sys 0.000s
I20250903 14:28:37.166877 31719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal
uuid: "3eee1ec72f6d487988d73e2e7c3b9023"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "ab651f82cceba880edce90e14d145a67"
server_key_iv: "3e5a9424f99cc229ca1c0b406a201340"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.166966 31719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:28:37.189994 31719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:28:37.193380 31719 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:28:37.193588 31719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:28:37.194208 31719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:28:37.195186 31719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:28:37.195247 31719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.195315 31719 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:28:37.195400 31719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.206066 31719 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:37439
I20250903 14:28:37.206440 31851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.129:37439 every 8 connection(s)
I20250903 14:28:37.207032 31719 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/data/info.pb
I20250903 14:28:37.211257 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 31719
I20250903 14:28:37.211375 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-0/wal/instance
I20250903 14:28:37.211592 19718 external_mini_cluster.cc:1468] Setting key 814f35a8e6c182aac7e4bacb673e704d
Sep 03 14:28:37 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20250903 14:28:37.219950 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:37.208503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:54823 (local address 127.19.65.190:32985)
0903 14:28:37.208767 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0903 14:28:37.208771 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 14:28:37.209212 (+ 441us) server_negotiation.cc:408] Connection header received
0903 14:28:37.210035 (+ 823us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:28:37.210038 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:28:37.210092 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:28:37.210166 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 14:28:37.211107 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:28:37.211789 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:28:37.213029 (+ 1240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:28:37.213213 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:28:37.215961 (+ 2748us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 14:28:37.215981 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 14:28:37.215986 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 14:28:37.216014 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 14:28:37.217834 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:28:37.218412 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:28:37.218418 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:28:37.218421 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:28:37.218476 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 14:28:37.218792 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 14:28:37.218796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 14:28:37.218797 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 14:28:37.218978 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 14:28:37.219043 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 14:28:37.219664 (+ 621us) server_negotiation.cc:300] Negotiation successful
0903 14:28:37.219777 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:28:37.221101 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:37.208729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:54823)
0903 14:28:37.209096 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:37.209134 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 14:28:37.209858 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:28:37.210312 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 14:28:37.210319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 14:28:37.210517 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0903 14:28:37.210978 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:28:37.210989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:28:37.211993 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 14:28:37.211995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 14:28:37.212887 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 14:28:37.212900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 14:28:37.213477 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 14:28:37.214201 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 14:28:37.214235 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 14:28:37.215816 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 14:28:37.217970 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:28:37.217977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:28:37.217988 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:28:37.218285 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:28:37.218583 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 14:28:37.218587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 14:28:37.218591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 14:28:37.218691 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 14:28:37.219140 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 14:28:37.219145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 14:28:37.219398 (+ 253us) client_negotiation.cc:769] Sending connection context
0903 14:28:37.220169 (+ 771us) client_negotiation.cc:240] Negotiation successful
0903 14:28:37.220398 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20250903 14:28:37.222143 31852 heartbeater.cc:344] Connected to a master server at 127.19.65.190:32985
I20250903 14:28:37.222435 31852 heartbeater.cc:461] Registering TS with master...
I20250903 14:28:37.222941 31852 heartbeater.cc:507] Master 127.19.65.190:32985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
I20250903 14:28:37.224252 31637 ts_manager.cc:194] Registered new tserver with Master: 3eee1ec72f6d487988d73e2e7c3b9023 (127.19.65.129:37439)
I20250903 14:28:37.225684 31637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:54823
Exception in thread "main" Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.19.65.190 from keytab /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20250903 14:28:37.244283 31711 server.cc:273] Received an EOF from the subprocess
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.130@KRBTEST.COM" created.
F20250903 14:28:37.244879 31699 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f628c3fcdcd google::LogMessage::Fail() at ??:0
@ 0x7f628c400b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f628c3fc7cc google::LogMessage::Flush() at ??:0
@ 0x7f628c3fdf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f628e0082d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f628e0050d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f628e009005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab.
@ 0x5605952e7364 std::function<>::operator()() at ??:0
I20250903 14:28:37.260468 19718 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1dJ511/build/debug/bin/kudu
/tmp/dist-test-task1dJ511/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:45265/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32985
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.65.148:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7f628d852924 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f628e0446db start_thread at ??:0
@ 0x7f628b5b371f clone at ??:0
W20250903 14:28:37.359858 31865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:28:37.360006 31865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:28:37.360051 31865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:28:37.362787 31865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 14:28:37.362838 31865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:28:37.362928 31865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20250903 14:28:37.366334 31865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.65.148:45265/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.65.190:32985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 14:12:21 UTC on 5fd53c4cbb9d
build id 7878
I20250903 14:28:37.367086 31865 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:28:37.367719 31865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:28:37.373054 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:28:37.373116 31865 server_base.cc:1047] running on GCE node
W20250903 14:28:37.373289 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:28:37.373056 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:28:37.373715 31865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:28:37.374153 31865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:28:37.375324 31865 hybrid_clock.cc:648] HybridClock initialized: now 1756909717375313 us; error 23 us; skew 500 ppm
Sep 03 14:28:37 dist-test-slave-4wpb krb5kdc[31134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 14:28:37.377471 31865 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20250903 14:28:37.378299 31865 webserver.cc:480] Webserver started at http://127.19.65.130:33111/ using document root <none> and password file <none>
I20250903 14:28:37.378746 31865 fs_manager.cc:362] Metadata directory not provided
I20250903 14:28:37.378799 31865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:28:37.378937 31865 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 14:28:37 dist-test-slave-4wpb krb5kdc[31134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756909717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20250903 14:28:37.402612 31865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data/instance:
uuid: "746722d43d77417cb4d661173fb406bd"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "5f159d38ac7291c5830ca53c002cf285"
server_key_iv: "76c987845c89d25035b5614350dc19b1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.403084 31865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "746722d43d77417cb4d661173fb406bd"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "5f159d38ac7291c5830ca53c002cf285"
server_key_iv: "76c987845c89d25035b5614350dc19b1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.406050 31865 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 14:28:37.417857 31887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.418854 31865 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 14:28:37.418946 31865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal
uuid: "746722d43d77417cb4d661173fb406bd"
format_stamp: "Formatted at 2025-09-03 14:28:37 on dist-test-slave-4wpb"
server_key: "5f159d38ac7291c5830ca53c002cf285"
server_key_iv: "76c987845c89d25035b5614350dc19b1"
server_key_version: "kuduclusterkey@0"
I20250903 14:28:37.419032 31865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:28:37.441241 31865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:28:37.443491 31865 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:28:37.443634 31865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:28:37.444078 31865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:28:37.444762 31865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:28:37.444801 31865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.444847 31865 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:28:37.444875 31865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:28:37.452090 31865 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43227
I20250903 14:28:37.452368 32000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.65.130:43227 every 8 connection(s)
I20250903 14:28:37.452967 31865 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/data/info.pb
I20250903 14:28:37.455607 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:37.454674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:47611)
0903 14:28:37.455016 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:37.455086 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
W20250903 14:28:37.455969 32001 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:32985 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:32985: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
I20250903 14:28:37.456396 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:37.456230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:36849)
0903 14:28:37.456314 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:37.456351 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20250903 14:28:37.456849 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:37.456711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:42905)
0903 14:28:37.456777 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:37.456807 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20250903 14:28:37.456959 32001 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250903 14:28:37.457707 19718 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1dJ511/build/debug/bin/kudu as pid 31865
I20250903 14:28:37.457801 19718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1dJ511/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756909585842267-19718-0/minicluster-data/ts-1/wal/instance
I20250903 14:28:37.458024 19718 external_mini_cluster.cc:1468] Setting key 753fb7128658bbefa9268f162a06d8af
I20250903 14:28:38.228050 31852 heartbeater.cc:499] Master 127.19.65.190:32985 was elected leader, sending a full tablet report...
I20250903 14:28:38.228807 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:38.228432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:57057)
0903 14:28:38.228649 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:38.228702 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
W20250903 14:28:38.229064 31852 heartbeater.cc:646] Failed to heartbeat to 127.19.65.190:32985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
I20250903 14:28:38.457752 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:38.457357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:56291)
0903 14:28:38.457631 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:38.457672 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":60,"threads_started":1}
I20250903 14:28:39.229897 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:39.229551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:39023)
0903 14:28:39.229770 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:39.229822 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":60,"threads_started":1}
I20250903 14:28:39.458654 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:39.458327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:49633)
0903 14:28:39.458535 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:39.458579 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20250903 14:28:40.231006 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:40.230585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:47525)
0903 14:28:40.230885 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:40.230932 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":68,"threads_started":1}
I20250903 14:28:40.459798 32010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:40.459409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:35931)
0903 14:28:40.459648 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:40.459695 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20250903 14:28:41.232102 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:41.231602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:46719)
0903 14:28:41.231972 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:41.232023 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":83,"threads_started":1}
I20250903 14:28:41.460927 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:41.460562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:39027)
0903 14:28:41.460793 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:41.460841 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20250903 14:28:42.233273 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:42.232890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.129:54683)
0903 14:28:42.233136 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:42.233192 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20250903 14:28:42.462024 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 14:28:42.461668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32985 (local address 127.19.65.130:40807)
0903 14:28:42.461900 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0903 14:28:42.461949 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
F20250903 14:28:42.462369 19718 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1756909722 (unix time) try "date -d @1756909722" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004d06) received by PID 19718 (TID 0x7ff476e4ad80) from PID 19718; stack trace: ***
@ 0x7ff48296c980 (unknown) at ??:0
@ 0x7ff48004efb7 gsignal at ??:0
@ 0x7ff480050921 abort at ??:0
@ 0x7ff4812bcdcd google::LogMessage::Fail() at ??:0
@ 0x7ff4812c0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff4812bc7cc google::LogMessage::Flush() at ??:0
@ 0x7ff4812bdf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff483f3dcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7ff483f3a9c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55f1776e1919 kudu::SecurityITest::StartCluster() at ??:0
@ 0x55f1776c9792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7ff4822c70ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ff4822bbbdb testing::Test::Run() at ??:0
@ 0x7ff4822bbd9d testing::TestInfo::Run() at ??:0
@ 0x7ff4822bc377 testing::TestSuite::Run() at ??:0
@ 0x7ff4822bc77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7ff4822c760d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ff4822bbe63 testing::UnitTest::Run() at ??:0
@ 0x55f177719c58 RUN_ALL_TESTS() at ??:0
@ 0x55f177718e45 main at ??:0
@ 0x7ff480031bf7 __libc_start_main at ??:0
@ 0x55f1776b106a _start at ??:0
2025-09-03 14:28:42.628 UTC [31583] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 14:28:42.629 UTC [31479] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 14:28:42.630 UTC [31475] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 14:28:42.632 UTC [31477] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 14:28:42.637 UTC [31476] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 14:28:42.638 UTC [31480] FATAL: terminating connection due to unexpected postmaster exit